Cloud Data Management %40 Yahoo%21

Feeding frenzy selectively materializing users%27 event feeds.

Benchmarking cloud serving systems with YCSB.
While the use of MapReduce systems (such as Hadoop) for large scale data analysis has been widely recognized and studied, we have recently seen an explosion in the number of systems developed for cloud data serving. These newer systems address “cloud OLTP ” applications, though they typically do not support ACID transactions. Examples of systems proposed for cloud serving use include BigTable, PNUTS, Cassandra, HBase, Azure, CouchDB, SimpleDB, Voldemort, and many others. Further, they are being applied to a diverse range of applications that differ considerably from traditional (e.g., TPC-C like) serving workloads. The number of emerging cloud serving systems and the wide range of proposed applications, coupled with a lack of applesto-apples performance comparisons, makes it difficult to understand the tradeoffs between systems and the workloads for which they are suited. We present the Yahoo! Cloud Serving Benchmark (YCSB) framework, with the goal of facilitating performance comparisons of the new generation of cloud data serving systems. We define a core set of benchmarks and report results for four widely used systems: Cassandra, HBase, Yahoo!’s PNUTS, and a simple sharded MySQL implementation. We also hope to foster the development of additional cloud benchmark suites that represent other classes of applications by making our benchmark tool available via open source. In this regard, a key feature of the YCSB framework/tool is that it is extensible—it supports easy definition of new workloads, in addition to making it easy to benchmark new systems.
Cloud Databases What%27s New%3F

Data Management in the Cloud.
Programme to provide a forum for exchanging and promoting ideas, experience and knowledge, as well as fostering collaborations among National Research and Education Networks and academic and research institutions on the topic of data storage, data management and cloud storage. The Task Force will be known as TF-Storage 2. The aims of this Task Force are: a. to provide a forum for exchanging and promoting ideas, experience and knowledge related to data storage, data management and cloud storage; b. to identify, and promote the use of common standards, techniques, (new) technology and procedures in the field of data storage that are relevant to the educational and research community; c. to be a focal point for (the gathering of) storage expertise within the European educational and research environment; d. to identify, coordinate and- where possible- collaborate on the development of storage services; e. to liaise with TF-EMC2 on the topic of using the NREN AAI federations with nonweb storage services; f. to liaise with SNIA and other communities such as the grid environment, HPC, the cloud environment, Internet2, APAN on storage and cloud activities. The Task Force will focus its activities on the TERENA constituency and as it recognizes there are many activities already under way in this constituency related to grids which have a storage and/or cloud component will work with these activities to prevent unnecessary duplication of effort. 3. The Task Force will be open to any individual who can offer appropriate expertise, manpower, equipment or services. Participation will be on a voluntary basis. 4. The Chair of the Task Force will be Jan Meijer (UNINETT) who will be responsible for preparing the agenda of each meeting, and for coordinating the work of the Task Force. He will also be responsible for ensuring that the agreed deliverables are produced. Sub groups may be formed as deemed appropriate. 5. The secretary of the Task Force will be appointed by TERENA. He/she will be responsible for taking the minutes at each meeting, and for making logistical arrangements as necessary.
A web of concepts.
this paper, we focus on the translation modeling problem. Before we turn to this problem, however, we should address an issue that may be a concern to some readers: Why do we estimate Pr(e) and Pr(fle) rather than estimate Pr(elf ) directly? We are really interested in this latter probability. Wouldn&#039;t we reduce our problems from three to two by this direct approach? If we can estimate Pr(fle) adequately, why can&#039;t we just turn the whole process around to estimate Pr(elf)? To understand this, imagine that we divide French and English strings into those that are well-formed and those that are ill-formed. This is not a precise notion. We have in mind that strings like Il va la bibliothque, or I live in a house, or even Colorless green ideas sleep furiously are well-formed, but that strings like lava I1 bibliothque or a I in live house are not. When we translate a French string into English, we can think of ourselves as springing from a well-formed French string into the sea of well-formed English strings with the hope of landing on a good one. It is important, therefore, that our model for Pr(elf ) concentrate its probability as much as possible on wellformed English strings. But it is not important that our model for Pr(fle ) concentrate its probability on well-formed French strings. If we were to reduce the probability of all well-formed French strings by the same factor, spreading the probability thus  265 liberated over ill-formed French strings, there would be no effect on our translations: the argument that maximizes some function f(x) also maximizes cf(x) for any positive constant c. As we shall see below, our translation models are prodigal, spraying probability all over the place, most of it on ill-formed French strings. In fact, as we discuss in Section 4.5, two...
Asynchronous view maintenance for VLSD databases.
The query models of the recent generation of very large scale distributed (VLSD) shared-nothing data storage systems, including our own PNUTS and others (e.g. BigTable, Dynamo, Cassandra, etc.) are intentionally simple, focusing on simple lookups and scans and trading query expressiveness for massive scale. Indexes and views can expand the query expressiveness of such systems by materializing more complex access paths and query results. In this paper, we examine mechanisms to implement indexes and views in a massive scale distributed database. For web applications, minimizing update latencies is critical, so we advocate deferring the work of maintaining views and indexes as much as possible. We examine the design space, and conclude that two types of view implementations, called remote view tables (RVTs) and local view tables (LVTs), provide a good tradeoff between system throughput and minimizing view staleness. We describe how to construct and maintain such view tables, and how they can be used to implement indexes, group-by-aggregate views, equijoin views and selection views. We also introduce and analyze a consistency model that makes it easier for application developers to cope with the impact of deferred view maintenance. An empirical evaluation quantifies the maintenance costs of our views, and shows that they can significantly improve the cost of evaluating complex queries.
Optimizing complex extraction programs over evolving text data.
Most information extraction (IE) approaches have considered only static text corpora, over which we apply IE only once. Many real-world text corpora however are dynamic. They evolve over time, and so to keep extracted information up to date we often must apply IE repeatedly, to consecutive corpus snapshots. Applying IE from scratch to each snapshot can take a lot of time. To avoid doing this, we have recently developed Cyclex, a system that recycles previous IE results to speed up IE over subsequent corpus snapshots. Cyclex clearly demonstrated the promise of the recycling idea. The work itself however is limited in that it considers only IE programs that contain a single IE“blackbox. ” In practice, many IE programs are far more complex, containing multiple IE blackboxes connected in a compositional “workflow.” In this paper, we present Delex, a system that removes the above limitation. First we identify many difficult challenges raised by Delex, including modeling complex IE programs for recycling purposes, implementing the recycling process efficiently, and searching for an optimal execution plan in a vast plan space with different recycling alternatives. Next we describe our solutions to these challenges. Finally, we describe extensive experiments with both rule-based and learning-based IE programs over two real-world data sets, which demonstrate the utility of our approach.
Optimizing complex extraction programs over evolving text data.
Most information extraction (IE) approaches have considered only static text corpora, over which we apply IE only once. Many real-world text corpora however are dynamic. They evolve over time, and so to keep extracted information up to date we often must apply IE repeatedly, to consecutive corpus snapshots. Applying IE from scratch to each snapshot can take a lot of time. To avoid doing this, we have recently developed Cyclex, a system that recycles previous IE results to speed up IE over subsequent corpus snapshots. Cyclex clearly demonstrated the promise of the recycling idea. The work itself however is limited in that it considers only IE programs that contain a single IE“blackbox. ” In practice, many IE programs are far more complex, containing multiple IE blackboxes connected in a compositional “workflow.” In this paper, we present Delex, a system that removes the above limitation. First we identify many difficult challenges raised by Delex, including modeling complex IE programs for recycling purposes, implementing the recycling process efficiently, and searching for an optimal execution plan in a vast plan space with different recycling alternatives. Next we describe our solutions to these challenges. Finally, we describe extensive experiments with both rule-based and learning-based IE programs over two real-world data sets, which demonstrate the utility of our approach.
Declarative networking.

The Claremont report on database research.
Abstract. Query processing is one of the most important mechanisms for data management, and there exist mature techniques for effective query optimization and efficient query execution. The vast majority of these techniques assume workloads of rather small transactional tasks with strong requirements for ACID properties. However, the emergence of new computing paradigms, such as grid and cloud computing, the increasingly large volumes of data commonly processed, the need to support data driven research, intensive data analysis and new scenarios, such as processing data streams on the fly or querying web services, the fact that the metadata fed to optimizers are often missing at compile time, and the growing interest in novel optimization criteria, such as monetary cost or energy consumption, create a unique set of new requirements for query processing systems. These requirements cannot be met by modern techniques in their entirety, although interesting solutions and efficient tools have already been developed for some of them in isolation. Next generation query processors are expected to combine features addressing all of these issues, and, consequently, lie at the confluence of several research initiatives. This paper aims to present a vision for such processors, to explain their functionality requirements, and to discuss the open issues, along with their challenges. 1
Building a Cloud for Yahoo%21

Bellwether analysis Searching for cost%2Deffective query%2Ddefined predictors in large databases.



Web Data Management.
The enormous growth in the number of documents circulated over the Web increases the need for improved Web data management systems. In order to evaluate the performance of these systems, various simulation approaches must be used. In this paper, we study the main simulation models that have been deployed for Web data management. More specifically, we survey the most recent simulation approaches, for Web data representation and storage (in terms of caching) as well as for Web data trace evaluation.
Cloud Computing at Yahoo%21

Parallel Evaluation of Composite Aggregate Queries.
Abstract — Aggregate measures summarizing subsets of data are valuable in exploratory analysis and decision support, especially when dependent aggregations can be easily specified and computed. A novel class of queries, called composite subset measures, was previously introduced to allow correlated aggregate queries to be easily expressed. This paper considers how to evaluate composite subset measure queries using a large distributed system. We describe a cross-node data redistribution strategy that takes into account the nested structure of a given query. The main idea is to group data into blocks in “cube space”, such that aggregations can be generated locally within each block, leveraging previously proposed optimizations per-block. The partitioning scheme allows overlap among blocks so that sliding window aggregation can be handled. Furthermore, it also guarantees that the final answer is the union of local results with no duplication and there is no need for the expensive data combination step. We identify the most important partitioning parameters and propose an optimization algorithm. We also demonstrate effectiveness of the optimizer to minimize the query response time. I.
Cloud Computing %2D Was Thomas Watson Right After All%3F

Efficient Information Extraction over Evolving Text Data.
Most current information extraction (IE) approaches have considered only static text corpora, over which we typically have to apply IE only once. Many real-world text corpora however are dynamic. They evolve over time, and to keep extracted information up to date, we often must apply IE repeatedly, to consecutive corpus snapshots. We describe Cyclex, an approach that efficiently executes such repeated IE, by recycling previous IE efforts. Specifically, given a current corpus snapshot U, Cyclex identifies text portions of U that also appears in the previous corpus snapshot V. Since Cyclex has already executed IE over V, it can now recycle the IE results of these parts, by combining these results with the results of executing IE over the remaining parts of U, to produce the complete IE results for U. Realizing Cyclex raises many challenges, including modeling information extractors, exploring the trade-off between runtime and completeness in identifying overlapping text, and making informed, cost-based decisions between redoing IE from scratch and recycling previous IE results. We describe initial solutions to these challenges, and experiments over two real-world data sets that demonstrate the utility of our approach. 1
Online Models for Content Optimization.
Address email We describe a new content publishing system that selects articles to serve to a user, choosing from an editorially programmed pool that is frequently refreshed. It is now deployed on a major Internet portal, and selects articles to serve to hundreds of millions of user visits per day, significantly increasing the number of user clicks over the original manual approach, in which editors periodically selected articles to display. Some of the challenges we face include a dynamic content pool, short article lifetimes, non-stationary click-through rates, and extremely high traffic volumes. The fundamental problem we must solve is to quickly identify which items are popular(perhaps within different user segments), and to exploit them while they remain current. We must also explore the underlying pool constantly to identify promising alternatives, quickly discarding poor performers. Our approach is based on tracking per article performance in near real time through online models. We describe the characteristics and constraints of our application setting, discuss our design choices, and show the importance and effectiveness of coupling online models with a simple randomization procedure. We discuss the challenges encountered in a production online content-publishing environment and highlight issues that deserve careful attention. Our analysis of this application also suggests a number of future research avenues. 1
Toward best%2Deffort information extraction.
Current approaches to develop information extraction (IE) programs have largely focused on producing precise IE results. As such, they suffer from three major limitations. First, it is often difficult to execute partially specified IE programs and obtain meaningful results, thereby producing a long “debug loop”. Second, it often takes a long time before we can obtain the first meaningful result (by finishing and running a precise IE program), thereby rendering these approaches impractical for time-sensitive IE applications. Finally, by trying to write precise IE programs we may also waste a significant amount of effort, because an approximate result – one that can be produced quickly – may already be satisfactory in many IE settings. To address these limitations, we propose iFlex, an IE approach that relaxes the precise IE requirement to enable best-effort IE. In iFlex, a developer U uses a declarative language to quickly write an initial approximate IE program P with a possible-worlds semantics. Then iFlex evaluates P using an approximate query processor to quickly extract an approximate result. Next, U examines the result, and further refines P if necessary, to obtain increasingly more precise results. To refine P, U can enlist a next-effort assistant, which suggests refinements based on the data and the current version of P. Extensive experiments on real-world domains demonstrate the utility of the iFlex approach.
Efficient bulk insertion into a distributed ordered table.
We study the problem of bulk-inserting records into tables in a system that horizontally range-partitions data over a large cluster of shared-nothing machines. Each table partition contains a contiguous portion of the table’s key range, and must accept all records inserted into that range. Examples of such systems include BigTable [8] at Google, and PNUTS [15] at Yahoo! During bulk inserts into an existing table, if most of the inserted records end up going into a small number of data partitions, the obtained throughput may be very poor due to ineffective use of cluster parallelism. We propose a novel approach in which a planning phase is invoked before the actual insertions. By creating new partitions and intelligently distributing partitions across machines, the planning phase ensures that the insertion load will be well-balanced. Since there is a tradeoff between the cost of moving partitions and the resulting throughput gain, the planning phase must minimize the sum of partition movement time and insertion time. We show that this problem is a variation of NP-hard bin-packing, reduce it to a problem of packing vectors, and then give a solution with provable approximation guarantees. We evaluate our approach on a prototype system deployed on a cluster of 50 machines, and show that it yields significant improvements over more naïve techniques.
Workload%2Daware anonymization techniques for large%2Dscale datasets.

Efficient top%2Dk processing over query%2Ddependent functions.
We study the efficient evaluation of top-k queries over data items, where the score of each item is dynamically computed by applying an item-specific function whose parameter value is specified in the query. For example, online retail stores rank items by price, which may be a function of the quantity being queried: “Stay 3 nights, get a 15 % discount on double-bed rooms. ” Similarly, while ranking possible routes in online maps by predicted congestion level, the score (congestion) is a function of the time being queried, e.g., “At 5PM on a Friday in Palo Alto, the congestion level on 101 North is high. ” Since the parameter—the number of nights or the time the online map is queried, in the above examples—is only known at query time, and online applications have stringent response-time requirements, it is infeasible to evaluate every item-specific function to determine the item scores, especially when the number of items is large. Further, space considerations make it infeasible to pre-compute and store the score of each item for each value of the input parameter. In this paper, we develop a novel technique that compresses the (large) set of item scores for all parameter values by dividing the parameter range into intervals, taking into account the expected query workload. This compressed representation is then used to do top-k pruning of query results. Our experiments show that the proposed techniques are scalable and efficient. 1.
PNUTS Yahoo%21%27s hosted data serving platform.
We describe PNUTS, a massively parallel and geographically distributed database system for Yahoo!’s web applications. PNUTS provides data storage organized as hashed or ordered tables, low latency for large numbers of concurrent requests including updates and queries, and novel per-record consistency guarantees. It is a hosted, centrally managed, and geographically distributed service, and utilizes automated load-balancing and failover to reduce operational complexity. The first version of the system is currently serving in production. We describe the motivation for PNUTS and the design and implementation of its table storage and replication layers, and then present experimental results. 1.
The Claremont report on database research.
Abstract. Query processing is one of the most important mechanisms for data management, and there exist mature techniques for effective query optimization and efficient query execution. The vast majority of these techniques assume workloads of rather small transactional tasks with strong requirements for ACID properties. However, the emergence of new computing paradigms, such as grid and cloud computing, the increasingly large volumes of data commonly processed, the need to support data driven research, intensive data analysis and new scenarios, such as processing data streams on the fly or querying web services, the fact that the metadata fed to optimizers are often missing at compile time, and the growing interest in novel optimization criteria, such as monetary cost or energy consumption, create a unique set of new requirements for query processing systems. These requirements cannot be met by modern techniques in their entirety, although interesting solutions and efficient tools have already been developed for some of them in isolation. Next generation query processors are expected to combine features addressing all of these issues, and, consequently, lie at the confluence of several research initiatives. This paper aims to present a vision for such processors, to explain their functionality requirements, and to discuss the open issues, along with their challenges. 1
Information extraction challenges in managing unstructured data.
Over the past few years, we have been trying to build an end-to-end system at Wisconsin to manage unstructured data, using extraction, integration, and user interaction. This paper describes the key information extraction (IE) challenges that we have run into, and sketches our solutions. We discuss in particular developing a declarative IE language, optimizing for this language, generating IE provenance, incorporating user feedback into the IE process, developing a novel wikibased user interface for feedback, best-effort IE, pushing IE into RDBMSs, and more. Our work suggests that IE in managing unstructured data can open up many interesting research challenges, and that these challenges can greatly benefit from the wealth of work on managing structured data that has been carried out by the database community. 1.
Purple SOX extraction management system.
We describe the Purple SOX (PSOX) EMS, a prototype Extraction Management System currently being built at Yahoo!. The goal of the PSOX EMS is to manage a large number of sophisticated extraction pipelines across different application domains, at the web scale and with minimum human involvement. Three key value propositions are described: extensibility, the ability to swap in and out extraction operators; explainability, the ability to track the provenance of extraction results; and social feedback support, the facility for gathering and reconciling multiple, potentially conflicting sources. 1.
Community Systems The World Online.

DBLife A Community Information Management Platform for the Database Research Community %28Demo%29.

Community Systems The World Online.

Source%2Daware Entity Matching A Compositional Approach.
Entity matching (a.k.a. record linkage) plays a crucial role in integrating multiple data sources, and numerous matching solutions have been developed. However, the solutions have largely exploited only information available in the mentions and employed a single matching technique. We show how to exploit information about data sources to significantly improve matching accuracy. In particular, we observe that different sources often vary substantially in their level of semantic ambiguity, thus requiring different matching techniques. In addition, it is often beneficial to group and match mentions in related sources first, before considering other sources. These observations lead to a large space of matching strategies, analogous to the space of query evaluation plans considered by a relational optimizer. We propose viewing entity matching as a composition of basic steps into a “match execution plan”. We analyze formal properties of the plan space, and show how to find a good match plan. To do so, we employ ideas from social network analysis to infer the ambiguity and relatedness of data sources. We conducted extensive experiments on several real-world data sets on the Web and in the domain of personal information management (PIM). The results show that our solution significantly outperforms current best matching methods. 1.
Optimizing mpf queries decision support and probabilistic inference.
Managing uncertain data using probabilistic frameworks has attracted much interest lately in the database literature, and a central computational challenge is probabilistic inference. This paper presents a broad class of aggregate queries, called MPF queries, inspired by the literature on probabilistic inference in statistics and machine learning. An MPF (Marginalize a Product Function) query is an aggregate query over a stylized join of several relations. In probabilistic inference, this join corresponds to taking the product of several probability distributions, while the aggregate operation corresponds to marginalization. Probabilistic inference can be expressed directly as MPF queries in a relational setting, and therefore, by optimizing evaluation of MPF queries, we provide scalable support for probabilistic inference in database systems. To optimize MPF queries, we build on ideas from database query optimization as well as traditional algorithms such as Variable Elimination and Belief Propagation from the probabilistic inference literature. Although our main motivation for introducing MPF queries is to support easy expression and efficient evaluation of probabilistic inference in a DBMS, we observe that this class of queries is very useful for a range of decision support tasks. We present and optimize MPF queries in a general form where arbitrary functions (i.e., other than probability distributions) are handled, and demonstrate their value for decision support applications through a number of illustrative and natural examples.
Databases on the web.

Databases on the web.

Declarative Information Extraction Using Datalog with Embedded Extraction Predicates.
In this paper we argue that developing information extraction (IE) programs using Datalog with embedded procedural extraction predicates is a good way to proceed. First, compared to current ad-hoc composition using, e.g., Perl or C++, Datalog provides a cleaner and more powerful way to compose small extraction modules into larger programs. Thus, writing IE programs this way retains and enhances the important advantages of current approaches: programs are easy to understand, debug, and modify. Second, once we write IE programs in this framework, we can apply query optimization techniques to them. This gives programs that, when run over a variety of data sets, are more efficient than any monolithic program because they are optimized based on the statistics of the data on which they are invoked. We show how optimizing such programs raises challenges specific to text data that cannot be accommodated in the current relational optimization framework, then provide initial solutions. Extensive experiments over real-world data demonstrate that optimization is indeed vital for IE programs and that we can effectively optimize IE programs written in this proposed framework. 1.
Declarative Information Extraction Using Datalog with Embedded Extraction Predicates.
In this paper we argue that developing information extraction (IE) programs using Datalog with embedded procedural extraction predicates is a good way to proceed. First, compared to current ad-hoc composition using, e.g., Perl or C++, Datalog provides a cleaner and more powerful way to compose small extraction modules into larger programs. Thus, writing IE programs this way retains and enhances the important advantages of current approaches: programs are easy to understand, debug, and modify. Second, once we write IE programs in this framework, we can apply query optimization techniques to them. This gives programs that, when run over a variety of data sets, are more efficient than any monolithic program because they are optimized based on the statistics of the data on which they are invoked. We show how optimizing such programs raises challenges specific to text data that cannot be accommodated in the current relational optimization framework, then provide initial solutions. Extensive experiments over real-world data demonstrate that optimization is indeed vital for IE programs and that we can effectively optimize IE programs written in this proposed framework. 1.
OLAP over Imprecise Data with Domain Constraints.
Several recent works have focused on OLAP over imprecise data, where each fact can be a region, instead of a point, in a multidimensional space. They have provided a multiple-world semantics for such data, and developed efficient solutions to answer OLAP aggregation queries over the imprecise facts. These solutions however assume that the imprecise facts can be interpreted independently of one another, a key assumption that is often violated in practice. Indeed, imprecise facts in real-world applications are often correlated, and such correlations can be captured as domain integrity constraints (e.g., repairs with the same customer names and models took place in the same city, or a text span can refer to a person or a city, but not both). In this paper we provide a solution to answer OLAP aggregation queries over imprecise data, in the presence of such domain constraints. We first describe a relatively simple yet powerful constraint language, and define what it means to take into account such constraints in query answering. Next, we prove that OLAP queries can be answered efficiently given a database D ∗ of fact marginals. We then exploit the regularities in the constraint space (captured in a constraint hypergraph) and the fact space to efficiently construct D*. Extensive experiments over real-world and synthetic data demonstrate the effectiveness of our approach.  
Building Structured Web Community Portals A Top%2DDown
Structured community portals extract and integrate information from raw Web pages to present a unified view of entities and relationships in the community. In this paper we argue that to build such portals, a top-down, compositional, and incremental approach is a good way to proceed. Compared to current approaches that employ complex monolithic techniques, this approach is easier to develop, understand, debug, and optimize. In this approach, we first select a small set of important community sources. Next, we compose plans that extract and integrate data from these sources, using a set of extraction/integration operators. Executing these plans yields an initial structured portal. We then incrementally expand this portal by monitoring the evolution of current data sources, to detect and add new data sources. We describe our initial solutions to the above steps, and a case study of employing these solutions to build DBLife, a portal for the database community. We found that DBLife could be built quickly and achieve high accuracy using simple extraction/integration operators, and that it can be maintained and expanded with little human effort. The initial solutions together with the case study demonstrate the feasibility and potential of our approach. 1.
Privacy Skyline Privacy with Multidimensional Adversarial Knowledge.
Privacy is an important issue in data publishing. Many organizations distribute non-aggregate personal data for research, and they must take steps to ensure that an adversary cannot predict sensitive information pertaining to individuals with high confidence. This problem is further complicated by the fact that, in addition to the published data, the adversary may also have access to other resources (e.g., public records and social networks relating individuals), which we call external knowledge. A robust privacy criterion should take this external knowledge into consideration. In this paper, we first describe a general framework for reasoning about privacy in the presence of external knowledge. Within this framework, we propose a novel multidimensional approach to quantifying an adversary’s external knowledge. This approach allows the publishing organization to investigate privacy threats and enforce privacy requirements in the presence of various types and amounts of external knowledge. Our main technical contributions include a multidimensional privacy criterion that is more intuitive and flexible than previous approaches to modeling background knowledge. In addition, we provide algorithms for measuring disclosure and sanitizing data that improve computational efficiency several orders of magnitude over the best known techniques. 1.
Exploratory mining in cube space.

Toward a PeopleWeb.
This paper introduces Metalog, a query/logical system designed to allow reasoning on the Web. Metalog tries to start filling in the so-called people axis, where the Web is tailored for the people, and not just for the machine. Besides allowing the formulation of declarative logical rules, Metalog&#039;s distinctive feature is to lower the entry access level, by employing a Pseudo Natural Language (PNL) interface, which is particularly easy to understand. This allows almost everybody to use Metalog, even without any particular expertize in the field. This capability, together with other advanced solutions that enhance customization and user friendliness, are key components for a wide adoption of intelligent semantic web technologies.
User%2DCentric Research Challenges in Community Information Management Systems.
In Cimple, a joint project between Wisconsin and Yahoo! Research, we are building systems that manage information for online communities. In this paper we discuss the fundamental roles users play in such systems, then the difficult user-centric research challenges raised by these roles, with respect to contributing to the system, accessing and using it, and leveraging the social interaction of users. 1
OLAP over uncertain and imprecise data.
We extend the OLAP data model to represent data ambiguity, specifically imprecision and uncertainty, and introduce an allocation-based approach to the semantics of aggregation queries over such data. We identify three natural query properties and use them to shed light on alternative query semantics. While there is much work on representing and querying ambiguous data, to our knowledge this is the first paper to handle both imprecision and uncertainty in an OLAP setting. 1


Learning from Aggregate Views.
In this paper, we introduce a new class of data mining problems called learning from aggregate views. In contrast to the traditional problem of learning from a single table of training examples, the new goal is to learn from multiple aggregate views of the underlying data, without access to the un-aggregated data. We motivate this new problem, present a general problem framework, develop learning methods for RFA (Restriction-Free Aggregate) views defined using COUNT, SUM, AVG and STDEV, and offer theoretical and experimental results that characterize the proposed methods. 1.
Toward a Query Language for Network Attack Data.
The growing sophistication and diversity of malicious activity in the Internet presents a serious challenge for network security analysts. In this paper, we describe our efforts to develop a database and query language for network attack data from firewalls, intrusion detection systems and honeynets. Our first step toward this objective is to develop a prototype database and query interface to identify coordinated scanning activity in network attack data. We have created a set of aggregate views and templatized SQL queries that consider timing, persistence, targeted services, spatial dispersion and temporal dispersion, thereby enabling us to evaluate coordinated scanning along these dimensions. We demonstrate the utility of the interface by conducting a case study on a set of firewall and intrusion detection system logs from Dshield.org. We show that the interface is able to identify general characteristics of coordinated activity as well as instances of unusual activity that would otherwise be difficult to mine from the data. These results highlight the potential for developing a more generalized query language for a broad class of network intrusion data. The case study also exposes some of the challenges we face in extending our system to more generalized queries over potentially vast quantities of data. 1
Exploratory Mining in Cube Space.

Workload%2Daware anonymization.
Data publishing generates much concern over the protection of individual privacy. Recent studies consider cases where the adversary may possess different kinds of knowledge about the data. In this article, we show that knowledge of the mechanism or algorithm of anonymization for data publication can also lead to extra information that assists the adversary and jeopardizes individual privacy. In particular, all known mechanisms try to minimize information loss and such an attempt provides a loophole for attacks. We call such an attack a minimality attack. In this article, we introduce a model called m-confidentiality which deals with minimality attacks, and propose a feasible solution. Our experiments show that minimality attacks are practical concerns on real datasets and that our algorithm can prevent such attacks with very little overhead and information loss. 
Relaxed%2Dcurrency serializability for middle%2Dtier caching and replication.

Managing information extraction state of the art and research directions.

Declarative networking language%2C execution and optimization.
The networking and distributed systems communities have recently explored a variety of new network architectures, both for applicationlevel overlay networks, and as prototypes for a next-generation Internet architecture. In this context, we have investigated declarative  networking: the use of a distributed recursive query engine as a powerful vehicle for accelerating innovation in network architectures [23, 24, 33]. Declarative networking represents a significant new application area for database research on recursive query processing. In this paper, we address fundamental database issues in this domain. First, we motivate and formally define the Network Datalog (NDlog) language for declarative network specifications. Second, we introduce and prove correct relaxed versions of the traditional semi-na ve query evaluation technique, to overcome fundamental problems of the traditional technique in an asynchronous distributed setting. Third, we consider the dynamics of network state, and formalize the &#034;eventual consistency&#034; of our programs even when bursts of updates can arrive in the midst of query execution. Fourth, we present a number of query optimization opportunities that arise in the declarative networking context, including applications of traditional techniques as well as new optimizations. Last, we present evaluation results of the above ideas implemented in our P2 declarative networking system, running on 100 machines over the Emulab network testbed.
Efficient Allocation Algorithms for OLAP Over Imprecise Data.
Recent work proposed extending the OLAP data model to support data ambiguity, specifically imprecision and uncertainty. A process called allocation was proposed to transform a given imprecise fact table into a form, called the Extended Database, that can be readily used to answer OLAP aggregation queries. In this work, we present scalable, efficient algorithms for creating the Extended Database (i.e., performing allocation) for a given imprecise fact table. Many allocation policies require multiple iterations over the imprecise fact table, and the straightforward evaluation approaches introduced earlier can be highly inefficient. Optimizing iterative allocation policies for large datasets presents novel challenges, and has not been considered previously to the best of our knowledge. In addition to developing scalable allocation algorithms, we present a performance evaluation that demonstrates their efficiency and compares their performance with respect to straightfoward approaches. 1.
Composite Subset Measures.
Measures are numeric summaries of a collection of data records produced by applying aggregation functions. Summarizing a collection of subsets of a large dataset, by computing a measure for each subset in the (typically, user-specified) collection is a fundamental problem. The multidimensional data model, which treats records as points in a space defined by dimension attributes, offers a natural space of data subsets to be considered as summarization candidates, and traditional SQL and OLAP constructs, such as GROUP BY and CUBE, allow us to compute measures for subsets drawn from this space. However, GROUP BY only allows us to summarize a limited collection of subsets, and CUBE summarizes all subsets in this space. Further, they restrict the measure used to summarize a data subset to be a one-step aggregation, using functions such as SUM, of field-values in the data records. In this paper, we introduce composite subset measures, computed by aggregating not only data records but also the measures of other related subsets. We allow summarization of naturally related regions in the multidimensional space, offering more flexibility than either GROUP BY or CUBE in the choice of what data subsets to summarize. Thus, our framework allows more meaningful summaries to be computed for a targeted collection of data subsets. We propose an algebra called AW-RA and an equivalent pictorial language called aggregation workflows. Aggregation workflows allow for intuitive expression of composite measure queries, and the underlying algebra is designed to facilitate efficient multi-scan execution. We describe an evaluation framework based on multiple passes of sorting and scanning over the original dataset. In each pass, several measures are evaluated simultaneously, and dependencies between these measures and containment relationships between the underlying subsets of data are orchestrated to reduce the memory footprint of the computation. We present a performance evaluation that demonstrates the benefits of our approach. 1.
Bellwether Analysis Predicting Global Aggregates from Local Regions.

Theory of nearest neighbors indexability.

Community Information Management.
We introduce Cimple, a joint project between the University of Illinois and the University of Wisconsin. Cimple aims to develop a software platform that can be rapidly deployed and customized to manage data-rich online communities. We first describe the envisioned working of such a software platform and our prototype, DBLife, which is a community portal being developed for the database research community. We then describe the technical challenges in Cimple and our solution approach. Finally, we discuss managing uncertainty and provenance, a crucial task in making our software platform practical. 1
Content%2C Metadata%2C and Behavioral Information Directions for Yahoo%21 Research.

Integrating DB and IR Technologies What is the Sound of One Hand Clapping%3F
Databases (DB) and information retrieval (IR) have evolved as separate fields. However, modern applications such as customer support, health care, and digital libraries require capabilities for both data and text management. In such settings, traditional DB queries, in SQL or XQuery, are not flexible enough to handle applicationspecific scoring and ranking. IR systems, on the other hand, lack efficient support for handling structured parts of the data and metadata, and do not give the application developer adequate control over the ranking function. This paper analyzes the requirements of advanced text- and data-rich applications for an integrated platform. The core functionality must be manageable, and the API should be easy to program against. A particularly important issue that we highlight is how to reconcile flexibility in scoring and ranking models with optimizability, in order to accommodate a wide variety of target applications efficiently. We discuss whether such a system needs to be designed from scratch, or can be incrementally built on top of existing architectures. The results of our analyses are cast into a series of challenges to the DB and IR communities.
Similarity search in high%2Ddimensional datasets.
Locality Sensitive Hashing (LSH) is a method of performing probabilistic dimension reduction of high-dimensional data. The basic idea is to hash the input items so that similar items are mapped to the same buckets with high probability (the number of buckets being much smaller than the universe of possible input items). This hashing scheme is widely used for the purposes of Image/Audio Similarity Identification and Near Duplicate Detection. A significant drawback of the basic LSH scheme is the requirement of a large number of hash tables to attain good search quality. Lv et al. have proposed a multi-probe LSH scheme which is built on the basic LSH indexing scheme and intends to counter the above drawback of the basic LSH scheme. The basic concept behind the multi-probe approach is that it searches in multiple buckets in parallel, which are likely to contain the query result. The reduction in space complexity can be attributed to the intelligent probing sequence followed by the Multi-Probe LSH scheme. We intend to take this multi-probe approach a step further by implementing a distributed version of the multi-probe approach described above. A distributed implementation would improve the time complexity as well as the space complexity by adding more computing power at our disposal. Thus our intended approach would guarantee scalability with respect to the real world scenario, without compromising on the efficiency of the multi-probe approach.
When Is Nearest Neighbors Indexable%3F
 Skyline queries are often used on data sets in multi-dimensional space for many decision-making applications. Traditionally, a point p is said to dominate another point q if, for all dimension, it is no worse than q and is better on at least one dimension. Therefore, the skyline of a data set consists of all points not dominated by any other point. To better cater to application requirements such as controlling the size of the skyline or handling data sets that are not well-structured, various works have been proposed to extend the definition of skyline based on variants of the dominance relationship. However, it is difficult to implement each of these variants separately in a system setting and instead effort must be made to provide a general framework so that these specific implementations can be easily materialized over the framework.
When Is Nearest Neighbors Indexable%3F
 Skyline queries are often used on data sets in multi-dimensional space for many decision-making applications. Traditionally, a point p is said to dominate another point q if, for all dimension, it is no worse than q and is better on at least one dimension. Therefore, the skyline of a data set consists of all points not dominated by any other point. To better cater to application requirements such as controlling the size of the skyline or handling data sets that are not well-structured, various works have been proposed to extend the definition of skyline based on variants of the dominance relationship. However, it is difficult to implement each of these variants separately in a system setting and instead effort must be made to provide a general framework so that these specific implementations can be easily materialized over the framework.
View Learning for Statistical Relational Learning With an Application to Mammography.
Statistical relational learning (SRL) constructs probabilistic models from relational databases. A key capability of SRL is the learning of arcs (in the Bayes net sense) connecting entries in different rows of a relational table, or in different tables. Nevertheless, SRL approaches currently are constrained to use the existing database schema. For many database applications, users find it profitable to define alternative “views ” of the database, in effect defining new fields or tables. Such new fields or tables can also be highly useful in learning. We provide SRL with the capability of learning new views. 1
View Learning for Statistical Relational Learning With an Application to Mammography.
Statistical relational learning (SRL) constructs probabilistic models from relational databases. A key capability of SRL is the learning of arcs (in the Bayes net sense) connecting entries in different rows of a relational table, or in different tables. Nevertheless, SRL approaches currently are constrained to use the existing database schema. For many database applications, users find it profitable to define alternative “views ” of the database, in effect defining new fields or tables. Such new fields or tables can also be highly useful in learning. We provide SRL with the capability of learning new views. 1
A Framework for Set%2DOriented Computation in Inductive Logic Programming and Its Application in Generalizing Inverse Entailment.

Declarative routing extensible routing with declarative queries.
The Internet&#039;s core routing infrastructure, while arguably robust and e#cient, has proven to be di#cult to evolve to accommodate the needs of new applications. Prior research on this problem has included new hard-coded routing protocols on the one hand, and fully extensible Active Networks on the other. In this paper, we explore a new point in this design space that aims to strike a better balance between the extensibility and robustness of a routing infrastructure. The basic idea of our solution, which we call declarative routing, is to express routing protocols using a database query language. We show that our query language is a natural fit for routing, and can express a variety of well-known routing protocols in a compact and clean fashion. We discuss the security of our proposal in terms of its computational expressive power and language design. Via simulation, and deployment on PlanetLab, we demonstrate that our system imposes no fundamental limits relative to traditional protocols, is amenable to query optimizations, and can sustain long-lived routes under network churn and congestion.
Incognito Efficient Full%2DDomain K%2DAnonymity.

The TEXTURE Benchmark Measuring Performance of Text Queries on a Relational DBMS.
We introduce a benchmark called TEXTURE  (TEXT Under RElations) to measure the relative  strengths and weaknesses of combining  text processing with a relational workload  in an RDBMS. While the well-known TREC  benchmarks focus on quality, we focus on e#-  ciency. TEXTURE is a micro-benchmark for  query workloads, and considers two central  text support issues that previous benchmarks  did not: (1) queries with relevance ranking,  rather than those that just compute all answers,  and (2) a richer mix of text and relational  processing, reflecting the trend toward  seamless integration. In developing this  benchmark, we had to address the problem  of generating large text collections that reflected  the (performance) characteristics of a  given &#034;seed&#034; collection; this is essential for a  controlled study of specific data characteristics  and their e#ects on performance. In addition  to presenting the benchmark, with performance  numbers for three commercial DBMSs,  we present and validate a synthetic generator  for populating text fields.
Caching with %27Good Enough%27 Currency%2C Consistency%2C and Completeness.
SQL extensions that allow queries to explicitly specify data quality requirements in terms of currency and consistency were proposed in an earlier paper. This paper develops a data quality-aware, finer grained cache model and studies cache design in terms of four fundamental properties: presence, consistency, completeness and currency. Such a model provides an abstract view of the cache to the query processing layer, and opens the door for adaptive cache management. We describe an implementation approach that builds on the MTCache framework for partially materialized views. The optimizer checks most consistency constraints and generates a dynamic plan that includes currency checks and inexpensive checks for dynamic consistency constraints that cannot be validated during plan compilation. Our solution not only supports transparent caching but also provides transactional fine grained data currency and consistency guarantees. 1.
OLAP Over Uncertain and Imprecise Data.
We extend the OLAP data model to represent data ambiguity, specifically imprecision and uncertainty, and introduce an allocation-based approach to the semantics of aggregation queries over such data. We identify three natural query properties and use them to shed light on alternative query semantics. While there is much work on representing and querying ambiguous data, to our knowledge this is the first paper to handle both imprecision and uncertainty in an OLAP setting. 1
Prediction Cubes.
Recently, “epitomes ” were introduced as patch-based probability models that are learned by compiling together a large number of examples of patches from input images. In this paper, we describe how epitomes can be used to model video data and we describe significant computational speedups that can be incorporated into the epitome inference and learning algorithm. In the case of videos, epitomes are estimated so as to model most of the small space-time cubes from the input data. Then, the epitome can be used for various modeling and reconstruction tasks, of which we show results for video super-resolution, video interpolation, and object removal. Besides computational efficiency, an interesting advantage of the epitome as a representation is that it can be reliably estimated even from videos with large amounts of missing data. We illustrate this ability on the task of reconstructing the dropped frames in video broadcast using only the degraded video and also in denoising a severely corrupted video. 1
Synopses for query optimization A space%2Dcomplexity perspective.
Database systems use precomputed synopses of data to estimate the cost of alternative plans during query optimization. A number of alternative synopsis structures have been proposed, but histograms are by far the most commonly used. While histograms have proved to be very effective in (cost estimation for) single-table selections, queries with joins have long been seen as a challenge; under a model where histograms are maintained for individual tables, a celebrated result of Ioannidis and Christodoulakis [1991] observes that errors propagate exponentially with the number of joins in a query. In this article, we make two main contributions. First, we study the space complexity of using synopses for query optimization from a novel information-theoretic perspective. In particular, we offer evidence in support of histograms for single-table selections, including an analysis over data distributions known to be common in practice, and illustrate their limitations for join queries. Second, for a broad class of common queries involving joins (specifically, all queries involving only key-foreign key joins) we show that the strategy of storing a small precomputed sample of the database yields probabilistic guarantees that are almost space-optimal, which is an important property if these samples are to be used as database statistics. This is the first such optimality
The EDAM project Mining atmospheric aerosol datasets.

Data Mining The Next Generation.
In this paper we identify a major area of research as a topic for next generation data mining. The research effort in the last decade on privacy preserving data mining has resulted in the development of numerous algorithms. However, most of the existing research has not been applied in any particular application context. Hence it is unclear whether the current algorithms are directly applicable in any particular problem context. In this paper we identify a significant application context that not only requires protection of privacy but also sophisticated data analysis. The area in question is supply chain management, arguably one of the most important research areas in production and operations management that has enormous practical relevance. We examine the area of supply chain management and identify research challenges and opportunities for privacy preserving data mining in the next generation. 1. New frontiers of privacy preserving data mining The area of privacy preserving data mining (PPDM) started with the seminal paper by [1] and prompted numerous research efforts since then. Although many fundamental questions still remain unanswered, the area has matured to the point where it must establish its relevance to larger societal needs, including those of businesses and industries. Most of the existing methods may need to be extended or modified before being directly applicable to real-world settings. In this paper we describe one such area, supply chain management (SCM) that encompasses the entire business processes of all industries that deal with products involving multiple interrelated trading partners. Although data mining is not part of every day tools that are applied in SCM, statistical data analysis for demand forecasting is an essential part of
The EDAM project mining mass spectra and more.

On the Integration of Structure Indexes and Inverted Lists.
Recently, there has been a great deal of interest in the development of techniques to evaluate path expressions over collections of XML documents. In general, these path expressions contain both structural and keyword components. Several methods have been proposed for processing path expressions over graph/tree-structured XML data. These methods can be classified into two broad classes. The first involves graph traversal where the input query is evaluated by traversing the data graph or some compressed representation. The other class involves information-retrieval style processing using inverted lists. In this framework, structure indexes have been proposed to be used as a substitute for graph traversal. These structure indexes are proven to be very effective when applied to queries that examine the “coarse ” structure of documents. For example, for many
Mass Spectrum Labeling Theory and Practice.
We introduce the problem of labeling a particle’s mass spectrum with the substances it contains, and develop several formal representations of the problem, taking into account practical complications such as unknown compounds and noise. This task is currently a bottle-neck in analyzing data from a new generation of instruments for real-time environmental monitoring. 1.
Mass Spectrum Labeling Theory and Practice.
We introduce the problem of labeling a particle’s mass spectrum with the substances it contains, and develop several formal representations of the problem, taking into account practical complications such as unknown compounds and noise. This task is currently a bottle-neck in analyzing data from a new generation of instruments for real-time environmental monitoring. 1.
Mass Collaboration A Case Study.
We present an overview of a novel customer support system developed at QUIQ between 1999 and 2003. The application was perhaps the first systematic use of mass collaboration, which builds on the observation that large communities of users can be effectively leveraged to help each other, and to advance the interests of the community as a whole. In recent years, mass collaboration has been proposed for uses such as information integration and even program debugging, and shows much promise. In this paper, we outline the main ideas and technical challenges, and describe the QUIQ architecture. Technically, the main achievements include a novel DB-IR engine; a scalable notification engine for a rich class of user-specified alerts; a powerful access control mechanism with support for roles, dynamic groups, and field-level access control; and techniques for editing navigation hierarchies in dynamic sites. 1.
Synopses for Query Optimization A Space%2DComplexity Perspective.
Database systems use precomputed synopses of data to estimate the cost of alternative plans during query optimization. A number of alternative synopsis structures have been proposed, but histograms are by far the most commonly used. While histograms have proved to be very effective in (cost estimation for) single-table selections, queries with joins have long been seen as a challenge; under a model where histograms are maintained for individual tables, a celebrated result of Ioannidis and Christodoulakis [1991] observes that errors propagate exponentially with the number of joins in a query. In this article, we make two main contributions. First, we study the space complexity of using synopses for query optimization from a novel information-theoretic perspective. In particular, we offer evidence in support of histograms for single-table selections, including an analysis over data distributions known to be common in practice, and illustrate their limitations for join queries. Second, for a broad class of common queries involving joins (specifically, all queries involving only key-foreign key joins) we show that the strategy of storing a small precomputed sample of the database yields probabilistic guarantees that are almost space-optimal, which is an important property if these samples are to be used as database statistics. This is the first such optimality
The EDAM Project Exploratory Data Analysis and Management at Wisconsin.

Cost%2DBased Labeling of Groups of Mass Spectra.

On the Integration of Structure Indexes and Inverted Lists.
Recently, there has been a great deal of interest in the development of techniques to evaluate path expressions over collections of XML documents. In general, these path expressions contain both structural and keyword components. Several methods have been proposed for processing path expressions over graph/tree-structured XML data. These methods can be classified into two broad classes. The first involves graph traversal where the input query is evaluated by traversing the data graph or some compressed representation. The other class involves information-retrieval style processing using inverted lists. In this framework, structure indexes have been proposed to be used as a substitute for graph traversal. These structure indexes are proven to be very effective when applied to queries that examine the “coarse ” structure of documents. For example, for many
Relaxed Currency and Consistency How to Say %22Good Enough%22 in SQL.
Despite the widespread and growing use of asynchronous copies to improve scalability, performance and availability, this practice still lacks a firm semantic foundation. Applications are written with some understanding of which queries can use data that is not entirely current and which copies are “good enough”; however, there are neither explicit requirements nor guarantees. We propose to make this knowledge available to the DBMS through explicit currency and consistency (C&amp;C) constraints in queries and develop techniques so the DBMS can guarantee that the constraints are satisfied. In this paper we describe our model for expressing C&amp;C constraints, define their semantics, and propose SQL syntax. We explain how C&amp;C constraints are enforced in MTCache, our prototype mid-tier database cache, including how constraints and replica update policies are elegantly integrated into the cost-based query optimizer. Consistency constraints are enforced at compile time while currency constraints are enforced at run time by dynamic plans that check the currency of each local replica before use and select sub-plans accordingly. This approach makes optimal use of the cache DBMS while at the same time guaranteeing that applications always get data that is &amp;quot;good enough &amp;quot; for their purpose. 1.
Support for Relaxed Currency and Consistency Constraints in MTCache.
A typical e-commerce site such as eBay makes frequent use of replicated and cached data. When browsing auctions in a category, the data (e.g., item prices, number of bids) may be a little out of date. However, most users understand and accept this,
Editorial.
calls &amp; calendarEDITORIAL
Editorial.
calls &amp; calendarEDITORIAL
The QUIQ Engine A Hybrid IR DB System.
For applications that involve rapidly changing textual data and also require traditional DBMS capabilities, current systems are unsatisfactory. In this paper, we describe a hybrid IR-DB system that serves as the basis for the QUIQConnect product, a collaborative customer support application. We present the novel query paradigm and system architecture, along with performance results. 1
Data Mining Fast Algorithms vs. Fast Results.

Guest editorial.
of vital signs, and medical emergency teams
Scaling mining algorithms to large databases.

A Framework for Measuring Differences in Data Characteristics.
A data mining algorithm builds a model that captures interesting aspects of the underlying  data. We develop a framework for quantifying the difference, called the deviation, between two  datasets in terms of the models they induce. In addition to being a quantitative, intuitively interpretable  measure of difference, the deviation between two datasets can also be computed very  fast. Our framework covers a wide variety of models including frequent itemsets, decision tree  classifiers, and clusters, and captures standard measures of deviation such as the misclassification  rate and the chi-squared metric as special cases. We also show how statistical techniques  can be applied to the deviation measure to assess whether the difference between two models    Contact author. Supported by a Microsoft Graduate Fellowship.  y  This research was supported by Grant 2053 from the IBM corporation.  z  Supported by Army Research Office grant DAAG55-98-1-0333.  1  is significant (i.e., whether the...
Mining Data Streams under Block Evolution.
Many scientific and commercial applications rely on an immediate processing of transient data streams. In addition to processing queries over streams, their continuative analysis has received more attention recently. Due to specific characteristics of data streams, common analysis techniques known from the area of data mining are not directly applicable to streams. One of the core operations in data analysis is density estimation, which is used for capturing an unknown distribution in various analysis tasks. Modern density estimates base either on kernel functions or wavelets, whereas the wavelet-based ones profit from their ability to identify discontinuities as well as local oscillations. In this paper, we present a new approach to computing wavelet density estimators over data streams. Our estimators allow continuous updates by arrival of new data and provide accurate analytical results, while consuming only a constant amount of memory. Moreover, our estimators are adaptive according to memory as well as CPU usage, i. e., we may change the memory size as well as its computing overhead at runtime. An experimental evaluation proves the feasibility of our approach and shows the superiority of wavelet density estimators compared to their kernel-based counterparts. 1
Mass collaboration and data mining.

DEMON Mining and Monitoring Evolving Data.
Data mining algorithms have been the focus of much research recently. In practice, the input data to  a data mining process resides in a large data warehouse whose data is kept up-to-date through periodic  or occasional addition and deletion of blocks of data. Most data mining algorithms have either assumed  that the input data is static, or have been designed for arbitrary insertions and deletions of data records.
Squeezing the Most out of Relational Database Systems.

DEMON Mining and Monitoring Evolving Data.
Data mining algorithms have been the focus of much research recently. In practice, the input data to  a data mining process resides in a large data warehouse whose data is kept up-to-date through periodic  or occasional addition and deletion of blocks of data. Most data mining algorithms have either assumed  that the input data is static, or have been designed for arbitrary insertions and deletions of data records.
Squeezing the Most out of Relational Database Systems.

Dynamic Histograms Capturing Evolving Data Sets.
In this paper, we introduce dynamic histograms, which are constructed and maintained incrementally.  We develop several dynamic histogram construction algorithms and show that  they come close to static histograms in quality. Our experimental study covers a wide range  of datasets and update patterns, including histogram maintenance in a shared-nothing environment.  Building upon the insights offered by the dynamic algorithms, we also propose a  new static histogram construction algorithm that is very fast and generates histograms that are  close in quality to the highly accurate (but expensive to construct!) V-Optimal histograms.  1 Introduction  The cost of executing a relational operator is a function of the sizes of the tuple streams that are input to the operator, which for intermediate operators are in turn determined by selectivities of the previous operators. The more complex a query is, the more important it is to have precise intermediate size estimates. Otherwise, errors in ...
From Browsing to Interacting DBMS Support for Responive Websites %28Abstract%29.

ICICLES Self%2DTuning Samples for Approximate Query Answering.
Approximate query answering systems provide  very fast alternatives to OLAP systems when applications  are tolerant to small errors in query answers.
Contrast Plots and P%2DSphere Trees Space vs. Time in Nearest Neighbour Searches.

RainForest %2D A Framework for Fast Decision Tree Construction of Large Datasets.
Classification of large datasets is an important data mining problem. Many classification algorithms have been proposed in the literature, but studies have shown that so far no algorithm uniformly outperforms all other algorithms in terms of quality. In this paper, we present a unifying framework for decision tree classifiers that separates the scalability aspects of algorithms for constructing a decision tree from the central features that determine the quality of the tree. This generic algorithm is easy to instantiate with specific algorithms from the literature (including C4.5, CART,
Guest Editor%27s Introduction Logic%2Dbased Heterogeneous Information Systems.

Clustering Large Datasets in Arbitrary Metric Spaces.
Clustering partitions a collection of objects into groups called clusters, such that similar objects fall into the same group. Similarity between objects is defined by a distance function satisfying the triangle inequality; this distance function along with the collection of objects describes a distance space. In a distance space, the only operation possible on data objects is the computation of distance between them. All scalable algorithms in the literature assume a special type of distance space, namely a k-dimensional vector space, which allows vector operations on objects.  We present two scalable algorithms designed for clustering very large datasets in distance spaces. Our first algorithm BUBBLE is, to our knowledge, the first scalable clustering algorithm for data in a distance space. Our second algorithm BUBBLE-FM improves upon BUBBLE by reducing the number of calls to the distance function, which may be computationally very expensive. Both algorithms make only a single scan o...
When Is %27%27Nearest Neighbor%27%27 Meaningful%3F
. We explore the effect of dimensionality on the &#034;nearest neighbor  &#034; problem. We show that under a broad set of conditions (much  broader than independent and identically distributed dimensions), as dimensionality  increases, the distance to the nearest data point approaches  the distance to the farthest data point. To provide a practical perspective,  we present empirical results on both real and synthetic data sets  that demonstrate that this effect can occur for as few as 10-15 dimensions.  These results should not be interpreted to mean that high-dimensional  indexing is never meaningful; we illustrate this point by identifying some  high-dimensional workloads for which this effect does not occur. However,  our results do emphasize that the methodology used almost universally  in the database literature to evaluate high-dimensional indexing  techniques is flawed, and should be modified. In particular, most such  techniques proposed in the literature are not evaluated versus simple...
Fast Density Estimation Using CF%2DKernel for Very Large Databases.

CACTUS %2D Clustering Categorical Data Using Summaries.
Clustering is an important data mining problem. Most of the earlier work on clustering focussed on numeric attributes which have a natural ordering on their attribute values. Recently, clustering data with categorical attributes, whose attribute values do not have a natural ordering, has received some attention. However, previous algorithms do not give a formal description of the clusters they discover and some of them assume that the user post-processes the output of the algorithm to identify the final clusters.  In this paper, we introduce a novel formalization of a cluster for categorical attributes by generalizing a definition of a cluster for numerical attributes. We then describe a very fast summarizationbased algorithm called CACTUS that discovers exactly such clusters in the data. CACTUS has two important characteristics. First, the algorithm requires only two scans of the dataset, and hence is very fast and scalable. Our experiments on a variety of datasets show that CACTUS outperforms previous work by a factor of  3  to  10.  Second, CACTUS can find clusters in subsets of all attributes and can thus perform a subspace clustering of the data. This feature is important if clusters do not span all attributes, a likely scenario if the number of attributes is very large. In a thorough experimental evaluation, we study the performance of CACTUS on real and synthetic datasets.  1 
Classification and Regression Money %2Acan%2A Grow on Trees.

A Framework for Measuring Changes in Data Characteristics.
A data mining algorithm builds a model that captures interesting aspects of the underlying data. We develop a framework for quantifying the difference, called the deviation, between two datasets in terms of the models they induce. Our framework covers a wide variety of models including frequent itemsets, decision tree classifiers, and clusters, and captures standard measures of deviation such as the misclassification rate and the chi-squared metric as special cases. We also show how statistical techniques can be applied to the deviation measure to assess whether the difference between two models is meaningful (i.e., whether the underlying datasets have statistically significant differences in their characteristics), and discuss several practical applications.
BOAT%2DOptimistic Decision Tree Construction.
Classification is an important data mining problem. Given a training database of records, each tagged with a class label, the goal of classification is to build a concise model that can be used to predict the class label of future, unlabeled records. A very popular class of classifiers are decision trees. All current algorithms to construct decision trees, including all main-memory algorithms, make one scan over the training database per level of the tree.  We introduce a new algorithm (BOAT) for decision tree construction that improves upon earlier algorithms in both performance and functionality. BOAT constructs several levels of the tree in only two scans over the training database, resulting in an average performance gain of 300% over previous work. The key to this performance improvement is a novel optimistic approach to tree construction in which we construct an initial tree using a small subset of the data and refine it to arrive at the final tree. We guarantee that any differen...
Bottom%2DUp Computation of Sparse and Iceberg CUBEs.
We introduce the Iceberg-CUBE problem as a reformulation of the datacube (CUBE) problem. The Iceberg-CUBE problem is to compute only those group-by partitions with an aggregate value (e.g., count) above some minimum support threshold. The result of Iceberg-CUBE can be used (1) to answer group-by queries with a clause such as HAVING COUNT(*) ?= X, where X is greater than the threshold, (2) for mining multidimensional association rules, and (3) to complement existing strategies for identifying interesting subsets of the CUBE for precomputation. We present a new algorithm (BUC) for Iceberg-CUBE computation. BUC builds the CUBE bottom-up; i.e., it builds the CUBE by starting from a group-by on a single attribute, then a group-by on a pair of attributes, then a group-by on three attributes, and so on. This is the opposite of all techniques proposed earlier for computing the CUBE, and has an important practical advantage: BUC avoids computing the larger group-bys that do not meet minimum sup...
Probabilistic Optimization of Top N Queries.

Review %2D Magic Sets and Other Strange Ways to Implement Logic Programs.

Mining Very Large Databases.

Equivalence of Keyed Relational Schemas by Conjunctive Queries.

Reminiscences on Influential Papers.

Database Management Systems.
ABSTRACT: In a high level query and data manipulation language such as SQL, requests are stated non-procedurally, without reference to access paths. This paper describes how System R chooses access paths for both simple (single relation) and complex queries (such as joins), given a user specification of desired data as a boolean expression of predicates. System R is an experimental database management system developed to carry out research on the relational model of data. System R was designed and built by members of the IBM San Jose Research&#039;Laboratory. 1.
Protecting the Quality of Service of Existing Information Systems.
Organizations that offer external access to their data need a mechanism that ensures a desired level of service for local users. We propose such a mechanism, called the provider agent (PA) architecture, that protects local users by ensuring a (DBA specified) quality of service for local requests in the face of computational demands made by external requests. The PA is a general purpose solution that enhances most information systems currently available. The novelty of our approach is the combination of request profiling with load control mechanisms to improve both protection and performance, while not requiring any modifications to the underlying information system. We demonstrate the effectiveness of the proposed techniques with a prototype PA for a commercial DBMS. 1. Introduction  People want access to information across organizational boundaries, and information systems are becoming increasingly accessible through integration technology. However, each system is usually owned by one...
Compressing Relations and Indexes.
We propose a new compression algorithm that is tailored to database applications. It can be applied to a collection of records, and is especially effective for records with many low to medium cardinality fields. In addition, this new technique supports very fast decompression. (Gzip typically takes 10 times the CPU time to decompress relations, while achieving about the same compression ratio.) Our algorithm achieves compression on numeric fields only; while low and medium cardinality fields of other types can be handled by mapping values to a small range of integers, our algorithms must be complemented by other compression techniques for general non-numeric (text, image) fields. Promising application domains include decision support systems (DSS), since `fact tables&#039;, which are by far the largest tables in these applications, contain many low and medium cardinality fields and typically no text fields [Kim96]. Further, our decompression rates are faster than typical disk throughputs fo...
SRQL Sorted Relational Query Language.
A relation is an unordered collection of records. Often, however, there is an underlying order (e.g., a sequence of stock prices), and users want to pose queries that reflect this order (e.g., find a weekly moving average). SQL provides no support for posing such queries. In this paper, we show how a rich class of queries reflecting sort order can be naturally expressed and efficiently executed with simple extensions to SQL. 1. Introduction  Ordered data, or sequences, can be found in a wide range of commercial, statistical, and scientific applications. These applications require DBMS support to store, manipulate, and query sequences efficiently, and such support is missing in RDBMSs since the relational model provides sets of tuples as its only data structure. SQL [2], the most widely used query language for relational systems is incapable of answering some common queries posed by commercial and scientific applications, such as moving aggregates. One approach that is being explored in...
RainForest %2D A Framework for Fast Decision Tree Construction of Large Datasets.
Classification of large datasets is an important data mining problem. Many classification algorithms have been proposed in the literature, but studies have shown that so far no algorithm uniformly outperforms all other algorithms in terms of quality. In this paper, we present a unifying framework for decision tree classifiers that separates the scalability aspects of algorithms for constructing a decision tree from the central features that determine the quality of the tree. This generic algorithm is easy to instantiate with specific algorithms from the literature (including C4.5, CART,
Processing Queries By Linear Constraints.
The emergence of several new application domains for databases has introduced the need for more efficient complex query handling than databases currently support. These application domains include On-Line Analytical Processing (OLAP),  Geographical Information Systems (GIS), and scientific databases. This paper focuses attention on a form of selection query, expressible in SQL but not evaluated efficiently by current DBMSs, with wide applicability in these new problem domains. We introduce a processing strategy for this class of queries, which we call queries by linear constraints (QBLC). This processing strategy can be implemented with a wide variety of multidimensional indexing structures that include the R-Tree variants, the k-d-B-Tree, the Buddy-Tree, and many more. Note that any processing strategy meant for general database use must guarantee that all correct answers are returned. Therefore, all numerical techniques we employ uphold this guarantee. Thus the most distinguishing ch...
Conjunctive Query Equivalence of Keyed Relational Schemas.
The notion of when two schemas are equivalent is fundamental to database design, schema integration, and data model translation. An important notion of schema equivalence, query equivalence was introduced in [2], and used to evaluate the correctness of schema transformations. The logically equivalent notion of calculous equivalence, as well as three progressively weaker notions of schema equivalence were introduced in 1984 by Hull [8, 9], who showed that two schemas with no dependencies are equivalent (under all four notions of equivalence) if and only if they are identical (up to renaming and re-ordering of attributes and relations). Hull also conjectured that the same result holds for schemas with primary keys. In this work, we resolve the conjecture in the affirmative for the case of query equivalence based on mappings using conjunctive relational queries with equality selections.  1 Introduction A fundamental concept in database theory is that of schema equivalence. Informally, tw...
Interactive Information Visualization.

DEVise Integrated Querying and Visualization of Large Datasets.

DEVise Integrated Querying and Visual Exploration of Large Datasets %28Demo Abstract%29.
)  M. Livny, R. Ramakrishnan, K. Beyer, G. Chen, D. Donjerkovic, S. Lawande, J. Myllymaki and K. Wenger  Department of Computer Sciences, University of Wisconsin--Madison 1210 W. Dayton St., Madison, Wisconsin 53706 Tel: (608)262-6611, Fax: (608)262-9777  fmiron,raghu,beyer,guangshu,donjerko,ssl,jussi,wengerg@cs.wisc.edu Abstract  DEVise is a data exploration system that allows users to easily develop, browse, and share visual presentations of large tabular datasets (possibly containing or referencing multimedia objects) from several sources. The DEVise framework, implemented in a tool that has been already successfully applied to a variety of real applications by a number of user groups, makes several contributions. In particular, it combines support for extended relational queries with powerful data visualization features. Datasets much larger than available main memory can be handled---DEVise is currently being used to visualize datasets well in excess of 100MB--- and data can be in...
The Case for Enhanced Abstract Data Types.
The explosion in complex multi-media content makes it crucial for database systems to support such data efficiently. We make the case that the next generation of object-relational database Data Type (E-ADT) technology, rather than on the “blackbox ” ADTs used in current systems. An E-ADT is an abstract data type that exposes the semantics of its methods. Query optimizations are performed using these semantics, resulting in efficient query processing. The added functionality does not compromise the modularity of data types and the extensibility of the type system. Fundamental architectural changes are required to build such a database system; these have been explored through the implementation of E-ADTs in Predutor. Initial performance results demonstrate an order of magnitude in performance improvements. 1
Rules in Data%2DBased Systems.
Relational database systems have been the backbone of business applications for more than 20 years. We promised to provide companies with a management information system that covers the core applications, including financials,
Constraints in Databases.
Introduction  Computer support for environmental monitoring involves complex data management. Data flows continually into the system and concurrently the data is analysed, in complex ways for validation, forecasting and the detection of significant events. This functionality can be supported by building a single interface on top of a combination of different systems - such as an object oriented database, a spatial database and an expert system. Unfortunately a hybrid system of this sort is very difficult to extend for new functionalities that may be required later (eg to support a new class of data). Moreover there are many technical problems which result in unexpected restrictions (eg in the management of concurrent updates). Constraint databases offer, within a single integrated database system, support for enhanced handling of data and knowledge. In the following we show the application of constraints for simpler more flexible data modelling and representation; for clean eff
Introduction to the Special Issue on Constraints and Databases.

BIRCH A New Data Clustering Algorithm and Its Applications.
. Data clustering is an important technique for exploratory data analysis, and has been studied for several years. It has been shown to be useful in many practical domains such as data classification and image processing. Recently, there has been a growing emphasis on exploratory analysis of very large datasets to discover useful patterns and/or correlations among attributes. This is called data mining, and data clustering is regarded as a particular branch. However existing data clustering methods do not adequately address the problem of processing large datasets with a limited amount of resources (e.g., memory and cpu cycles). So as the dataset size increases, they do not scale up well in terms of memory requirement, running time, and result quality. In this paper, an efficient and scalable data clustering method is proposed, based on a new in-memory data structure called CF-tree, which serves as an in-memory summary of the data distribution. We have implemented it in a system called...
BIRCH An Efficient Data Clustering Method for Very Large Databases.
Finding useful patterns in large datasets has attracted considerable interest recently, and one of the most widely st,udied problems in this area is the identification of clusters, or deusel y populated regions, in a multi-dir nensional clataset. Prior work does not adequately address the problem of large datasets and minimization of 1/0 costs. This paper presents a data clustering method named Bfll (;”H (Balanced Iterative Reducing and Clustering using Hierarchies), and demonstrates that it is especially suitable for very large databases. BIRCH incrementally and clynamicall y clusters incoming multi-dimensional metric data points to try to produce the best quality clustering with the available resources (i. e., available memory and time constraints). BIRCH can typically find a goocl clustering with a single scan of the data, and improve the quality further with a few aclditioual scans. BIRCH is also the first clustering algorithm proposerl in the database area to handle “noise) ’ (data points that are not part of the underlying pattern) effectively. We evaluate BIRCH’S time/space efficiency, data input order sensitivity, and clustering quality through several experiments. We also present a performance comparisons of BIR (;’H versus CLARA NS, a clustering method proposed recently for large datasets, and S11OW that BIRCH is consistently 1
Cost%2DBased Optimization for Magic Algebra and Implementation.
Magic sets rewriting is a well-known optimization heuristic for complex decision-support queries. There can be many variants of this rewriting even for a single query, which differ greatly in execution performance. We propose cost-based techniques for selecting an efficient variant from the many choices. Our first contribution is a practical scheme that modelsmagic sets rewriting as a special join method that can be added to any cost-based query optimizer. We derive cost formulas that allow an optimizer to choose the best variant of the rewriting and to decide whether it is beneficial. The order of complexity of the optimization process is preserved by limiting the search space in a reasonable manner. We have implemented this technique in IBM&#039;s DB2 C/S V2 database system. Our performance measurements demonstrate that the costbasedmagic optimization technique performs well, and that without it, several poor decisions could be made. Our second contribution is a formal algebraic model of ...
On the Computation of Multidimensional Aggregates.
At the heart of all OLAP or multidimensional data analysis applications is the ability to simultaneously aggregate across many sets of dimensions. Computing multidimensional aggregates is a performance bottleneck for these applications. This paper presents fast algorithms for computing a collection of groupbys. We focus on a special case of the aggregation problem -- computation of the CUBE operator. The CUBE operator requires computing group-bys on all possible combinations of a list of attributes, and is equivalent to the union of a number of standard group-by operations. We show howthe structure of CUBE computation can be viewed in terms of a hierarchy of group-by operations. Our algorithms extend sort-based and hash-based grouping methods with several optimizations, like combining common operations across multiple group-bys, caching, and using pre-computed group-bys for computing other group-bys. Empirical evaluation shows that the resulting algorithms give much better performance compared to straightforward methods. This paper combines work done concurrently on computing the data cube by two different teams as reported in [SAG96] and [DANR96].  
The Design and Implementation of a Sequence Database System.
This paper discusses the design and implementation of SEQ, a database system with support for sequence data. SEQ models a sequence as an ordered collection of records, and supports a declarative sequence query language based on an algebra of query operators, thereby permitting algebraic query optimization and evaluation. SEQ has been built as a component of the PREDATOR database system that provides support for relational and other kinds of complex data as well. There are three distinct contributions made in this paper. (1) We describe the specification of sequence queries using the SEQUIN query language. (2) We quantitatively demonstrate the importance of various storage and optimization techniques by studying their effect on performance. (3) We present a novel nested design paradigm used in PREDATOR to combine sequence ‘and relational data.
Rules in Database Systems.
Database researchers have recognized that integrating a production rules facility into a database system provides a uniform mechanism for a number of advanced database features including integrity constraint enforcement, derived data maintenance, triggers, protection, version control, and others. In addition, a database system with rule processing capabilities provides a useful platform for large and efficient knowledge-base and expert systems. Database systems with production rules are referred to as active database systems, and the field of active database systems has indeed been active. This paper summarizes current work in active database systems and suggests future research directions. Topics covered include database rule languages, rule processing semantics, and implementation issues.  1 Introduction  Database systems provide persistent storage for massive amounts of data and powerful interfaces for querying and modifying this data. Even so, most database systems are passive, si...
Constraints in Databases.
Introduction  Computer support for environmental monitoring involves complex data management. Data flows continually into the system and concurrently the data is analysed, in complex ways for validation, forecasting and the detection of significant events. This functionality can be supported by building a single interface on top of a combination of different systems - such as an object oriented database, a spatial database and an expert system. Unfortunately a hybrid system of this sort is very difficult to extend for new functionalities that may be required later (eg to support a new class of data). Moreover there are many technical problems which result in unexpected restrictions (eg in the management of concurrent updates). Constraint databases offer, within a single integrated database system, support for enhanced handling of data and knowledge. In the following we show the application of constraints for simpler more flexible data modelling and representation; for clean eff
Some Promising Directions for Database Research.

Magic Conditions.
Much recent work has focussed on the bottomup evaluation of Datalog programs. One approach, called magic-sets, is based on rewriting a logic program so that bottom-up fixpoint evaluation of the program avoids generation of irrelevant facts ([BMSU86, BR87, Ram88]). It is widely believed that the principal application of the magicsets technique is to restrict computation in recursive queries using equijoin predicates. We extend the magic-sets transformation to use predicates other than equality (X ? 10, for example). This Extended Magic-Set technique has practical utility in &#034;real&#034; relational databases, not only for recursive queries, but for non-recursive queries as well; in ([MFPR90]) we use the results in this paper and those in [MPR90] to define a magic-set transformation for relational databases supporting SQL and its extensions, going on to describe an implementation of magic in Starburst ([HFLP89]). We also give preliminary performance measurements. In extending magic-sets, we des...
E%2DADTs Turbo%2DCharging Complex Data.
The next generation of database applications will be dominated by rich and complex data types. The ADT technology of today&#039;s object-relational database systems cannot provide adequate performance for such applications. The basic stumbling block is the lack of semantic knowledge provided to the database system about the ADT operations. We are developing novel &#034;Enhanced ADT&#034; (E-ADT) technology that overcomes this problem, while retaining the extensibility of the database system. In effect, this &#034;turbo-charges&#034; the complex data types, resulting in dramatically improved performance. E-ADT technology is being demonstrated on several kinds of multi-media data in the PREDAT OR database system at Cornell.  1 Introduction  We are witnessing an explosion in the volume and complexity of digital information that people want to access and analyze. Much of this data is &#034;multi-media&#034; like images, video, audio, and documents. Several other kinds of complex data are also important, including geographic...
Data Modeling and Querying in the PIQ Image DBMS.
Image Database Management Systems (IDBMS) aim to store large collections of images, and to support efficient content-based retrieval. In this paper we explore the idea of image data modeling as a tool for describing image domains, with the twofold objectives of guiding feature extraction and incorporating semantics into the extracted summaries of images. We discuss the implementation of these ideas in the PIQ image DBMS, and demonstrate that significat gains  in the expressiveness of content-based queries can be achieved.  Image Database Management Systems (IDBMS) aim to store large collections of images, and to support efficient content-based retrieval of the images. Like any other DBMS, an IDBMS has a storage facility, the data is arranged according to some data model, and it has a query facility.  The main IDBMS issue this paper explores is the data model and its impact on the expressiveness of the query language. The kind of queries we focus on are selection queries. In other words...
A Framework for Testing Safety and Effective Computability.
Constraint Logic Programming (CLP) is a merger of two declarative paradigms: constraint solving and logic programming. Although a relatively new field, CLP has progressed in several quite different directions. In particular, the early fundamental concepts have been adapted to better serve in different areas of applications. In this survey of CLP, a primary goal is to give a systematic description of the major trends in terms of common fundamental concepts. The three main parts cover the theory, implementation issues, and programming for applications.
Applications of Logic Databases

SEQ A Model for Sequence Databases.
This paper presents the SEQ model which is the basis for a system to manage various kinds of sequence data. The model separates the data from the ordering information, and includes operators based on two distinct abstractions of a sequence. The main contributions of the SEQ model are: (a) it can deal with different types of sequence data, (b) it supports an expressive range of sequence queries, (c) it draws from many of the diverse existing approaches to modeling sequence data.  1 Introduction  Data over ordered domains such as time or linear position often reflect the underlying order, and can be naturally viewed as a sequence. Applications involving sequence data have typically led to a specialized model and a stylized language. In this paper, we present the SEQ model which is the basis for a system we are building to manage and manipulate diverse kinds of sequence data. It is reasonable for a database user to ask, &#034;What kinds of queries can I ask on sequence data, and how can I expr...
Scientific Journals Extinction or Explosion%3F %28Panel%29.

Containment of Conjunctive Queries Beyond Relations as Sets

Space Optimization in Deductive Databases.
In the bottom-up evaluation of logic programs and recursively defined views on databases, all generated facts are usually assumed to be stored until the end of the evaluation. Discarding facts during the evaluation, however, can considerably improve the efficiency of the evaluation: the space needed to evaluate the program, the I/O costs, the costs of maintaining and accessing indices, and the cost of eliminating duplicates may all be reduced. Given an evaluation method that is sound, complete and does not repeat derivation steps, we consider how facts can be discarded during the evaluation without compromising these properties. We show that every such space optimization method has three components, the first to ensure soundness and completeness, the second to avoid redundancy (i.e., repetition of derivations), and the third to reduce `fact lifetimes&#039; (i.e., the time period for which each fact must be retained during evaluation). We present new techniques based on providing bounds on t...
A survey of deductive database systems.
 

Argument Reduction by Factoring.
We identify a useful property of a program with re-spect to a predicate, called factoring. While we prove that detecting factorability is undecidable in general, we show that for a large class of programs, the pro-gram obtained by applying the Magic Sets transfor-mation is factorable with respect to the recursive predicate. When the factoring property holds, a sim-ple optimization of the program generated by the Magic Sets transformation results in a new program that is never less efficient than the Magic Sets pro-gram and is often dramatically more efficient, due to the reduction of the arity of the recursive predi-cate. We show that the concept of factoring general-izes some previously identified special cases of recur-sions, including separable recursions and right- and left-linear recursions, and that the specialized evalua-tion algorithms and rewriting strategies developed for
Sequence Queries and Applications.
This paper studies aggregate search in transaction time databases. Specifically, each object in such a database can be modeled as a horizontal segment, whose y-projection is its search key, and its x-projection represents the period when the key was valid in history. Given a query timestamp qt and a key range �qk, a count-query retrieves the number of objects that are alive at qt, and their keys fall in �qk. We provide a method that accurately answers such queries, with error less than 1 ε + ε · Nalive(qt), where Nalive(qt) is the number of objects alive at time qt, and ε is any constant in (0,1]. Denoting the disk page size as B, and n = N/B, our technique requires O(n) space, processes any query in O(logB n) time, and supports each update in O(logB n) amortized I/Os. As demonstrated by extensive experiments, the proposed solutions guarantee query results with extremely high precision (median relative error below 5%), while consuming only a fraction of the space occupied by the existing approaches that promise precise results. To appear in VLDB Journal.
Schema Equivalence in Heterogeneous Systems Bridging Theory and Practice %28Extended Abstract%29.

Constraint Objects.

Sequence Query Processing.
Many applications require the ability to manipulate sequences of data. We motivate the importance of sequence query processing, and present a framework for the optimization of sequence queries based on several novel techniques. These include query transformations, optimizations that utilize meta--data, and caching of intermediate results. We present a bottom-up algorithm that generates an efficient query evaluation plan based on cost estimates. This work also identifies a number of directions in which future research can be directed.  1 Introduction  Many real life applications manipulate data that is inherently sequential. Such data is logically viewed and queried in terms of a sequence abstraction and is often physically stored as a sequence. Databases should (a) allow sequences to be queried in a declarative manner, utilizing the ordered semantics of the data, and (b) take advantage of the opportunities available for query optimization. Relational databases are inadequate in this re...
A Performance Study of Transitive Closure Algorithms.


Semantics and Optimization of Constraint Queries in Databases.
Chimera is a novel database language jointly designed by researchers at Politecnico di Milano, Bonn University, and ECRC of Munich in the context of the Esprit Project P6333, IDEA.  Chimera integrates an object-oriented data model, a declarative query language based on deductive rules, and an active rule language for reactive processing. In this paper, we present the authors&#039; proposal for constraint management in Chimera, which relies on the declarative specification of passive constraints as deductive rules and on their transformation into different types of active rules.  1 Introduction  Constraints are declarative specifications of properties that must be satisfied in any database instance. In OODBs constraints are typically used to restrict the admissible extension of a class, to impose restrictions on the state of individual objects, or to specify general laws that must be obeyed by sets of objects possibly belonging to different classes. To face violations of a constraint by an u...
Rule Ordering in Bottom%2DUp Fixpoint Evaluation of Logic Programs.
Abstract Logic programs can be evaluated bottom-up by repeatedly applying all rules, in &amp;quot;iterations&amp;quot;, until the fixpoint is reached. However, it is often desirable--and in some cases, e.g. programs with stratified negation, even necessary to guarantee the semantics--to apply the rules in some order. We present two algorithms that apply rules in a specified order without repeating inferences. One of them (GSN) is capable of dealing with a wide range of rule orderings but with a little more overhead than the well-known semi-naive algorithm (which we call BSN). The other (PSN) handles a smaller class of rule orderings, but with no overheads beyond those in BSN. We also demonstrate that by choosing a good ordering, we can reduce the number of rule applications (and thus joins). We present a theoretical analysis of rule orderings and identify orderings that minimize the number of rule applications (for all possible instances of the base relations) with respect to a class of orderings called fair orderings. We also show that while non-fair orderings may do a little better on some data sets, they can do much worse on others. The analysis is supplemented by performance results.
Schema equivalence in heterogeneous systems bridging theory and practice.
Current theoretical work offers measures of schema equivalence based on the information capacity of schemas. This work is based on the existence of abstract functions satisfying various restrictions between the sets of all instances of two schemas. In considering schemas that arise in practice, however, it is not clear how to reason about the existence of such abstract functions. Further, these notions of equivalence tend to be too liberal in that schemas are often considered equivalent when a practitioner would consider them to be different. As a result, practical integration methodologies have not utilized this theoretical foundation and most of them have relied on ad-hoc approaches. We present results that seek to bridge this gap. First, we consider the problem of deciding information capacity equivalence and dominance of schemas that occur in practice, i.e., those that can express inheritance and simple integrity constraints. We show that this problem is undecidable. This undecidab...
How to Forget the Past Without Repeating It.

Abstract Interpretation of Logic Programs using Magic Transformations.

The CORAL Deductive System
CORAL is a deductive system which supports a rich declarative language, and an interface to C++ which  allows for a combination of declarative and imperative programming. The declarative query language supports  general Horn clauses augmented with complex terms, set-grouping, aggregation, negation, and relations with  tuples that contain (universally quantified) variables. A CORAL declarative program can be organized as a  collection of interacting modules. The CORAL implementation supports a wide range of evaluation strategies,  and automatically chooses an efficient evaluation strategy for each module in the program. In addition, users  are permitted to guide query optimization, if desired, by selecting from among a wide range of control choices  at the level of each module.  The CORAL system provides imperative constructs such as update, insert and delete rules. CORAL  also has an interface with C++, and users can program in a combination of declarative CORAL and C++  extended with ...
Explaining Program Execution in Deductive Systems.
. Programs in deductive database and programming systems have a natural meaning that is based upon their mathematical reading as logical rules. High-level `explanations&#039; of a program evaluation/execution can be constructed to provide added functionality: (1) To debug a program by following a chain of deductions leading to an unexpected (and possibly incorrect) conclusion; (2) To follow the derivation of certain  correct conclusions to determine why and how they are reached; (3) To identify consequences of a (typically, incorrect or unexpected) fact. This functionality can be utilized either to perform post-mortem analysis of a session, or to interactively develop programs by running queries and viewing their deductions simultaneously. `Explanations&#039; of programs are especially important in the context of deductive databases for three reasons: (1) These programs could involve recursion, and hence, the chain of inferences is often not evident. (2) When the input data set is large, it is v...
Optimizations of Bottom%2DUp Evaluation with Non%2DGround Terms.

Extending the Well%2DFounded and Valid Semantics for Aggregation.
We present a very general technique for defining semantics for programs that use aggregation. We use the technique to extend the well-founded semantics and the valid semantics, both of which were designed to provide semantics for programs with negation, to handle programs that contain possibly recursive use of aggregation. The generalization is based on a simple but powerful idea of aggregation on three-valued multisets. The use of three-valued multisets makes our extended well-founded semantics, which we call aggregate-well-founded semantics, more intuitive than the extension of well-founded models by Van Gelder [Van92]. Our semantics and Van Gelder&#039;s semantics agree on many programs, and on others our semantics provides results that Van Gelder says are intuitive and desirable, but that his semantics does not provide. The extended valid semantics, which we call the aggregate-valid semantics, generalizes the intuition behind several semantics presented earlier for aggregation;...
Explaining Program Execution in Deductive Systems.
. Programs in deductive database and programming systems have a natural meaning that is based upon their mathematical reading as logical rules. High-level `explanations&#039; of a program evaluation/execution can be constructed to provide added functionality: (1) To debug a program by following a chain of deductions leading to an unexpected (and possibly incorrect) conclusion; (2) To follow the derivation of certain  correct conclusions to determine why and how they are reached; (3) To identify consequences of a (typically, incorrect or unexpected) fact. This functionality can be utilized either to perform post-mortem analysis of a session, or to interactively develop programs by running queries and viewing their deductions simultaneously. `Explanations&#039; of programs are especially important in the context of deductive databases for three reasons: (1) These programs could involve recursion, and hence, the chain of inferences is often not evident. (2) When the input data set is large, it is v...
Understanding Schemas.
Before the problem of schema integration and translation can be adequately addressed, a precise understanding of schemas is needed. We present an analysis of the notion of schema as used by existing integration methodologies. We show how inherent ambiguities and imprecision in traditional definitions of schema can hamper the development of formal schema integration methodologies. Specifically, traditional notions of schema contain data in the form of metadata, as well as superfluous structuring information that is not semantically meaningful. We argue that it is important to cleanly separate structural information from data, and remove from consideration artifacts of a specific data model or design methodology. To this end, we introduce the notion of a schema intension to capture the semantic content of a schema.  1 A Closer look at Schemas  Current work on schema integration uses the notion of schema as defined by a specific data model or design methodology. These notions vary in such...
Implementation of the CORAL Deductive Database System.
  CORAL is a deductive database system that provides a modular, declarative query language/programming language. CORAL is a deductive system that supports a rich declarative language, provides a wide range of evaluation methods, and allows a combination of declarative and imperative programming. The data can be persistent on disk or can reside in main-memory. We describe the architecture and implementation of CORAL. There were two important goals in the design of the CORAL architecture: (1) to integrate the different optimization techniques in a reasonable fashion, and (2) to allow users to influence the evaluation strategies used so as to exploit the full power of the CORAL implementation. A CORAL declarative program can be organized as a collection of interacting modules and this module structure is the key to satisfying both these goals. The high level module interface allows modules with different evaluation techniques to interact in a transparent fashion. Further, users can optionally tailor the execution of a program by selecting from among a wide range of control choices at the level of each module. CORAL also has an interface with C++, and users can program in a combination of declarative CORAL, and C++ extended with CORAL primitives. A high degree of extensibility is provided by allowing C++ programmers to use the class structure of C++ to enhance the CORAL implementation.
The CORAL Deductive Database System.

The Use of Information Capacity in Schema Integration and Translation.
In this paper, we carefully explore the assumptions behind using information capacity equivalence as a measure of correctness for judging transformed schemas in schema integration and translation methodologies. We present a classification of common integration and translation tasks based on their operational goals and derive from them the relative information capacity requirements of the original and transformed schemas. We show that for many tasks, information capacity equivalence of the schemas is not strictly required. Based on this, we present a new definition of correctness that reflects each undertaken task. We then examine existing methodologies and show how anomalies can arise when using those that do not meet the proposed correctness criteria.  1 Introduction  Formal work on schema equivalence has largely been ignored within practical schema integration and translation tools. Practitioners have felt that theoretical work is too narrow in scope to be applicable to the problems ...
Coral   Adding Object%2DOrientation to a Logic Database Language.
Coral++ is a database programming language that integrates Coral [23] with the C++ type system. The data model allows arbitrary C++ objects in database facts, and the declarative query language extends Coral with C++ expressions in rules. Coral++ also supports an imperative rule-based sub-language that is integrated with C++, providing support for updates. The design and implementation of Coral++ incorporates several important decisions: the data model is based on C++, and class definitions and method invocations are handled entirely by the C++ compiler; the notion of classes is kept orthogonal to the re- The work of Raghu Ramakrishnan, Divesh Srivastava and Praveen Seshadri was supported by a David and Lucile Packard Foundation Fellowship in Science and Engineering, a Presidential Young Investigator Award with matching grants from DEC, Tandem and Xerox, and NSF grant IRI-9011563. The addresses of the authors are Computer Sciences Department, University of Wisconsin, Madison, WI 53706,...
MIMSY A System for Stock Market Analysis.

The CORAL Deductive Database System.

MIMSY A System for Analyzing Time Series Data in the Stock Market Domain.
vi 1 Overview 1 1.1 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 1.2 Acknowledgements : : : : : : : : : : : : : : : : : : : : : : : : : : : 5 2 The Language 6 2.1 Overview of the Query Language : : : : : : : : : : : : : : : : : : : 7 2.2 The Command Line Interface : : : : : : : : : : : : : : : : : : : : : 8 2.3 Graphing Utilities : : : : : : : : : : : : : : : : : : : : : : : : : : : : 10 2.4 The Select Clause : : : : : : : : : : : : : : : : : : : : : : : : : : : : 13 2.4.1 Select Attributes : : : : : : : : : : : : : : : : : : : : : : : : 13 2.4.2 The Repeated Clause : : : : : : : : : : : : : : : : : : : : : : 14 2.5 The When Clause : : : : : : : : : : : : : : : : : : : : : : : : : : : : 15 2.5.1 Relational Operator Clauses : : : : : : : : : : : : : : : : : : 15 2.5.2 Change Clauses : : : : : : : : : : : : : : : : : : : : : : : : : 16 2.5.3 Crosses Clauses : : : : : : : : : : : : : : : : : : : : : : : : : 17 2.5.4 Condition Interval : : : : : : : : : : : ...
Transitive Closure Algorithms Based on Graph Traversal.

Pushing Constraint Selections.
this paper, we present a procedure that generates and propagates minimum QRP-constraints (if it terminates), based on the definition and uses of program predicates (Section 4). By propagating minimum QRP-constraints to the original program, we obtain a program that fully utilizes the constraint information present in the original program. This procedure is based on two sub-procedures: 1. Procedure Gen Prop predicate-constraints, which generates and propagates constraints that are satisfied by program predicates based on their definitions.
Logical Query Optimization by Proff%2DTree Transformation.

Pushing Constraint Selections.
this paper, we present a procedure that generates and propagates minimum QRP-constraints (if it terminates), based on the definition and uses of program predicates (Section 4). By propagating minimum QRP-constraints to the original program, we obtain a program that fully utilizes the constraint information present in the original program. This procedure is based on two sub-procedures: 1. Procedure Gen Prop predicate-constraints, which generates and propagates constraints that are satisfied by program predicates based on their definitions.
Controlling the Search in Bottom%2DUp Evaluation.
Bottom-up evaluation of queries on deductive databases has many advantages over an evaluation scheme such as Prolog. It is sound and complete with respect to the declarative semantics of least Herbrand models for positive Horn clause programs. In particular, it is able to avoid infinite loops by detecting repeated (possibly cyclic) subgoals. Further, in many database applications, it is more efficient than Prolog due to its set-orientedness. However, the completely set-oriented, breadth-first search strategy of bottomup evaluation has certain disadvantages. For example, to evaluate several classes of programs with negation (or aggregation), it is necessary to order the inferences; in essence, we must evaluate all answers to a negative subgoal before making an inference that depends upon the negative subgoal. A completely breadth-first search strategy ([14]) would have to maintain a lot of redundant subgoal dependency information to achieve this. We present a technique to order the use ...
Pushing Constraint Selections.
this paper, we present a procedure that generates and propagates minimum QRP-constraints (if it terminates), based on the definition and uses of program predicates (Section 4). By propagating minimum QRP-constraints to the original program, we obtain a program that fully utilizes the constraint information present in the original program. This procedure is based on two sub-procedures: 1. Procedure Gen Prop predicate-constraints, which generates and propagates constraints that are satisfied by program predicates based on their definitions.
The Valid Model Semantics for Logic Programs.
We present the valid model semantics, a new approach to providing semantics for logic programs with negation, setterms and grouping. The valid model semantics is a threevalued semantics, and is defined in terms of a `normal form&#039; computation. The valid model semantics also gives meaning to the generation and use of non-ground facts (i.e., facts with variables) in a computation. The formulation of the semantics in terms of a normal form computation offers important insight not only into the valid model semantics, but also into other semantics proposed earlier. We show that the valid model semantics extends the well-founded semantics in a natural manner, and has several advantages over it. The well-founded semantics can also be understood using a variant of the normal form computations that we use; the normal form computations used for valid semantics seem more natural than those used for well-founded semantics. We also show that the valid model semantics has several other desirable prop...
CORAL %2D Control%2C Relations and Logic.
CORAL is a modular declarative query language/programming language that supports general Horn clauses with complex terms, set-grouping, aggregation, negation, and relations with tuples that contain (universally quantified) variables. Support for persistent relations is provided by using the EXODUS storage manager. A unique feature of CORAL is that it provides a wide range of evaluation strategies and allows users to --- optionally --- tailor execution of a program through high-level annotations. A CORAL program is organized as a collection of modules,  and this structure is used as the basis for expressing control choices. CORAL has an interface to C++, and uses the class structure of C++ to provide extensibility. Finally, CORAL supports a command sublanguage, in which statements are evaluated in a user-specified order. The statements can be queries, updates, production-system style rules, or any command that can be typed in at the CORAL system prompt.   This work was supported by a Da...
Optimizations of Bottom%2DUp Evaluation with Non%2DGround Terms %28Extended Abstract%29.

Bottom%2DUp Evaluation of Logic Programs.
 In recent years, much work has been directed towards evaluating logic programs and queries on deductive databases by using an iterative bottom-up fixpoint computation. The resulting techniques offer an attractive alternative to Prolog-style top-down evaluation in several situations. They are sound and complete for positive Horn clause programs, are well-suited to applications with large volumes of data (facts), and can support a variety of extensions to the standard logic programming paradigm.  We present the basics of database query evaluation and logic programming evaluation, and then discuss bottom-up fixpoint evaluation. We discuss an approach based upon using a program transformation (&#034;Magic Templates&#034;) to restrict search, followed by fixpoint computation using a technique (&#034;Semi-naive evaluation&#034;) that avoids repeated inferences. The program transformation technique focuses the fixpoint evaluation, which is a forward-chaining strategy, by propagating bindings in the goal in a ma...
Top%2DDown versus Bottom%2DUp Revisited.

Space Optimization in the Bottom%2DUp Evaluation of Logic Programs.
In the bottom-up evaluation of a logic program, all generated facts are usually assumed to be stored until the end of the evaluation. Considerable gains can be achieved by instead discarding facts that are no longer required: the space needed to evaluate the program is reduced, I/O costs may be reduced, and the costs of maintaining and accessing indices, eliminating duplicates etc. are reduced. Thus, discarding facts early could achieve time as well as space improvements. Given an evaluation method that is sound, complete and does not repeat derivation steps, we consider how facts can be discarded during the evaluation without compromising these properties. Our first contribution is to show that such a space optimization technique has three distinct components. Informally, we must make all derivations that we can with each fact, detect all duplicate derivations of facts and try to order the computation so as to minimize the &#034;lifespan &#034; of each fact. This separation enables us to use di...
Aggregation and Relevance in Deductive Databases.
In this paper we present a technique to optimize queries on deductive databases that use aggregate operations such as min, max, and “largest Ic values. ” Our approach is based on an extended notion of relevance of facts to queries that takes aggregate operations into account. The approach has two parts: a rewriting part that labels predica.tes with “ag-gregate selections, ” and an evaluat,ion part. t.hat, makes use of “aggregate selections ” to detect that facts are irrelevant and discards them. The rewriting complements standard rewrit-ing algorithms like Magic sets, and the evaluation essentially refines Semi-Naive evaluation. 1
Parallelism in Logic Programs.
Parallelizing logic programming has attracted much interest in the research community, because of the intrinsic OR- and AND-parallelisms of logic programs. One research stream aims at transparent exploitation of parallelism in existing logic programming languages such as Prolog, whale the family of concurrent logic languages develops language constructs allowing programmers to express the concurrency—that is, the communication and synchronization between parallel processes—within their algorithms. This article concentrates mainly on transparent exploitation of parallelism and surveys the most mature solutions to the problems to be solved in order to obtain efficient implementations. These solutions have been implemented, and the most efficient parallel logic programming systems reach effective speedups over state-of-the-art sequential Prolog implementations. The article also addresses current and prospective research issues in extending the applicability and the efficiency of existing systems, such as models merging the transparent parallehsm and the concurrent logic languages approaches, combination of constraint logic programming with parallelism, and use of highly parallel architectures.
On the Power of Magic.

Magic Templates A Spellbinding Approach To Logic Programs.
We consider a bottom-up query-evaluation scheme in which facts of relations are allowed to have nonground terms. The Magic Sets query-rewriting technique is generalized to allow arguments of predicates to be treated as bound even though the rules do not provide ground bindings for those arguments. In particular, we regard as &#034;bound&#034; any argument containing a function symbol or a variable that appears more than once in the argument list. Generalized &#034;magic &#034; predicates are thus defined to compute the set of all goals reached in a top-down exploration of the rules, starting from a given query goal; these goals are not facts of constants as in previous versions of the Magic Sets algorithm. The magic predicates are then used to restrict a bottom-up evaluation of the rules so that there are no redundant actions; that is, every step of the bottom-up computation must be performed by any algorithm that uses the same sideways information passing strategy (sips). The price paid, compared to prev...
Magic Conditions.
Much recent work has focussed on the bottomup evaluation of Datalog programs. One approach, called magic-sets, is based on rewriting a logic program so that bottom-up fixpoint evaluation of the program avoids generation of irrelevant facts ([BMSU86, BR87, Ram88]). It is widely believed that the principal application of the magicsets technique is to restrict computation in recursive queries using equijoin predicates. We extend the magic-sets transformation to use predicates other than equality (X ? 10, for example). This Extended Magic-Set technique has practical utility in &#034;real&#034; relational databases, not only for recursive queries, but for non-recursive queries as well; in ([MFPR90]) we use the results in this paper and those in [MPR90] to define a magic-set transformation for relational databases supporting SQL and its extensions, going on to describe an implementation of magic in Starburst ([HFLP89]). We also give preliminary performance measurements. In extending magic-sets, we des...
Parallelism in Logic Programs.
Parallelizing logic programming has attracted much interest in the research community, because of the intrinsic OR- and AND-parallelisms of logic programs. One research stream aims at transparent exploitation of parallelism in existing logic programming languages such as Prolog, whale the family of concurrent logic languages develops language constructs allowing programmers to express the concurrency—that is, the communication and synchronization between parallel processes—within their algorithms. This article concentrates mainly on transparent exploitation of parallelism and surveys the most mature solutions to the problems to be solved in order to obtain efficient implementations. These solutions have been implemented, and the most efficient parallel logic programming systems reach effective speedups over state-of-the-art sequential Prolog implementations. The article also addresses current and prospective research issues in extending the applicability and the efficiency of existing systems, such as models merging the transparent parallehsm and the concurrent logic languages approaches, combination of constraint logic programming with parallelism, and use of highly parallel architectures.
Magic is Relevant.
We define the magic-sets transformation for traditional relational systems (with duplicates, aggregation and grouping), as well as for relational systems extended with recursion. We compare the magic-sets rewriting to traditional optimization techniques for nonrecursive queries, and use performance experiments to argue that the magic-sets transformation is often a better optimization technique.  1 Introduction  &#034;Magic-sets&#034; is the name of a query transformation algorithm ([BMSU86]) (and now a class of algorithms    Part of this work was done at the IBM Almaden Research Center. Work at Stanford was supported by an NSF grant IRI87 -22886, an Air Force grant AFOSR-88-0266, and a grant of IBM Corporation.  y  Author&#039;s current affiliation: Tandem Computers.  z  Part of this work was done while the author was visiting IBM Almaden Research Center. Work at Wisconsin was supported by an IBM Faculty Development Award and an NSF grant IRI-8804319.   --- Generalized Magic-sets of [BR87], Magic Tem...
The Magic of Duplicates and Aggregates.
We present a formal treatment of multisets (that arise when duplicates are not eliminated) and aggregate operators for deductive and relational databases. We define the semantics rigorously and extend the magic-sets technique to programs containing multisets and aggregates. The work presented here is an important step in demonstrating the applicability of the magic-sets technique for optimizing queries in commercial query languages such as SQL.  1 Introduction  Previous treatments of Datalog and proposed extensions have treated a program as a collection of definitions of  sets of facts (tuples). On the other hand, commercial query languages such as SQL typically support the definition of sets and multisets of tuples, and provide aggregate operators such as SUM and COUNT over sets and multisets. The ability to deal with multisets    Part of this work was done at the IBM Almaden Research Center. Work at Stanford was supported by an NSF grant IRI87 -22886, an Air Force grant AFOSR-88-0266...
How to Forget the Past Without Repeating It.

Rule Ordering in Bottom%2DUp Fixpoint Evaluation of Logic Programs.
Abstract Logic programs can be evaluated bottom-up by repeatedly applying all rules, in &amp;quot;iterations&amp;quot;, until the fixpoint is reached. However, it is often desirable--and in some cases, e.g. programs with stratified negation, even necessary to guarantee the semantics--to apply the rules in some order. We present two algorithms that apply rules in a specified order without repeating inferences. One of them (GSN) is capable of dealing with a wide range of rule orderings but with a little more overhead than the well-known semi-naive algorithm (which we call BSN). The other (PSN) handles a smaller class of rule orderings, but with no overheads beyond those in BSN. We also demonstrate that by choosing a good ordering, we can reduce the number of rule applications (and thus joins). We present a theoretical analysis of rule orderings and identify orderings that minimize the number of rule applications (for all possible instances of the base relations) with respect to a class of orderings called fair orderings. We also show that while non-fair orderings may do a little better on some data sets, they can do much worse on others. The analysis is supplemented by performance results.
CORAL %2D A Database Programming Language.

Bounds on the Propagation of Selection into Logic Programs.

D%E9j%E0 Vu in Fixpoints of Logic Programs.

Proof%2DTree Transformation Theorems and Their Applications.

Efficient Evaluation of Right%2D%2C Left%2D%2C and Mult%2DLineare Rules.

Argument Reduction by Factoring.
We identify a useful property of a program with re-spect to a predicate, called factoring. While we prove that detecting factorability is undecidable in general, we show that for a large class of programs, the pro-gram obtained by applying the Magic Sets transfor-mation is factorable with respect to the recursive predicate. When the factoring property holds, a sim-ple optimization of the program generated by the Magic Sets transformation results in a new program that is never less efficient than the Magic Sets pro-gram and is often dramatically more efficient, due to the reduction of the arity of the recursive predi-cate. We show that the concept of factoring general-izes some previously identified special cases of recur-sions, including separable recursions and right- and left-linear recursions, and that the specialized evalua-tion algorithms and rewriting strategies developed for
Magic Templates A Spellbinding Approach to Logic Programs.
We consider a bottom-up query-evaluation scheme in which facts of relations are allowed to have nonground terms. The Magic Sets query-rewriting technique is generalized to allow arguments of predicates to be treated as bound even though the rules do not provide ground bindings for those arguments. In particular, we regard as &#034;bound&#034; any argument containing a function symbol or a variable that appears more than once in the argument list. Generalized &#034;magic &#034; predicates are thus defined to compute the set of all goals reached in a top-down exploration of the rules, starting from a given query goal; these goals are not facts of constants as in previous versions of the Magic Sets algorithm. The magic predicates are then used to restrict a bottom-up evaluation of the rules so that there are no redundant actions; that is, every step of the bottom-up computation must be performed by any algorithm that uses the same sideways information passing strategy (sips). The price paid, compared to prev...
An Axiomatic Approach to Deciding Query Safety in Deductive Databases.

Optimizing Existential Datalog Queries.

A Framework for Testing Safety and Effective Computability of Extended Datalog %28Extended Abstract%29.

Efficient Transitive Closure Algorithms.
We present a new transitive closure algorithm that is based on strong component detection. The new algorithm is more efficient than the previous transitive closure algorithms that are based on strong components detection, since it does not generate unnecessary partial successor sets and scans the input graph only once. Keywords: Design of algorithms; transitive closure; strong components; Tarjan&#039;s algorithm; random graphs; simulation  1 Introduction We consider a directed graph G = (V; E), where V is the set of vertices and E ` V \Theta V is the set of edges. Without loss of generality, we assume that G has no self loops (v; v). The graph is represented as a set of adjacency lists Adj(v) = fw j (v; w) 2 Eg. The transitive closure of G is a graph G  +  = (V; E  +  ) such that for all v; w 2 V there is an edge (v; w) 2 E  +  if and only if there is a non-null path from v to w in G. The  successor set of a vertex v in G is the set Succ(v) = fw j (v; w) 2 E  +  g. Two vertices v and w in ...
Performance Evaluation of Data Intensive Logic Programs.

Sets and Negation in a Logic Database Language %28LDL1%29.

Bounds on the Propagation of Selection into Logic Programs.

On the Power of Magic.

Safety of Recursive Horn Clauses With Infinite Relations.

Annotations for Distributed Programming in Logic.

An Amateur%27s Introduction to Recursive Query Processing Strategies.

An Amateur%27s Introduction to Recursive Query Processing Strategies.



Safety of Recursive Horn Clauses With Function Symbols.

The MR Diagram %2D A Model for Conceptual Database Design.

